{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Chapter 11\n",
    "## Model Evaluation\n",
    "\n",
    "### 11.0 Introduction\n",
    "### 11.1 Cross-Validating Models\n",
    "#### Problem\n",
    "You want to evaluate how well your model will work in the real world\n",
    "\n",
    "#### Solution\n",
    "Create a pipeline that preprocesses the data, trains the model, and then evaluates it using cross-validation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.964931719428926"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# load libraries\n",
    "from sklearn import datasets, metrics\n",
    "from sklearn.model_selection import KFold, cross_val_score\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "# load digits dataset\n",
    "digits = datasets.load_digits()\n",
    "\n",
    "# create features matrix\n",
    "features = digits.data\n",
    "\n",
    "# create target vector\n",
    "target = digits.target\n",
    "\n",
    "# create standardizer\n",
    "standardizer = StandardScaler()\n",
    "\n",
    "# create logitic regression object\n",
    "logit = LogisticRegression()\n",
    "\n",
    "# create a pipeline that standardizes, then runs logistic regression\n",
    "pipeline = make_pipeline(standardizer, logit)\n",
    "\n",
    "# create k-fold cross-validation\n",
    "kf = KFold(n_splits=10, shuffle=True, random_state=1)\n",
    "\n",
    "# conduct k-fold cross-validation\n",
    "cv_results = cross_val_score(pipeline, # Pipeline\n",
    "                             features, # feature matrix\n",
    "                             target, # target vector\n",
    "                             cv=kf, # cross-validation technique,\n",
    "                             scoring=\"accuracy\", # loss function\n",
    "                             n_jobs=-1) # use all CPU cores\n",
    "\n",
    "# calculate mean\n",
    "cv_results.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Discussion\n",
    "#### See Also\n",
    "* Why every statistician should know about cross-validation (https://robjhyndman.com/hyndsight/crossvalidation/)\n",
    "* Cross-Validation Gone Wrong (http://betatim.github.io/posts/cross-validation-gone-wrong/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 11.2 Creating a Baseline Regression Model\n",
    "#### Problem\n",
    "You want a simple baseline regression model to compare against your model\n",
    "#### Solution\n",
    "Use scikit-learn's DummyRegressor to create a simple model to use as a baseline:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.001119359203955339"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# load libraries\n",
    "from sklearn.datasets import load_boston\n",
    "from sklearn.dummy import DummyRegressor\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# load data\n",
    "boston = load_boston()\n",
    "\n",
    "# create features\n",
    "features, target = boston.data, boston.target\n",
    "\n",
    "# make test and training split\n",
    "features_train, features_test, target_train, target_test = train_test_split(features, target, random_state=0)\n",
    "\n",
    "# create a dummy regressor\n",
    "dummy = DummyRegressor(strategy='mean')\n",
    "\n",
    "# \"Train\" dummy regressor\n",
    "dummy.fit(features_train, target_train)\n",
    "\n",
    "# Get R-squared score\n",
    "dummy.score(features_test, target_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To compare, we train our model and evaluate the performance score:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6353620786674623"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# load library\n",
    "from sklearn.linear_model import LinearRegression\n",
    "\n",
    "# train simple linear regression model\n",
    "ols = LinearRegression()\n",
    "ols.fit(features_train, target_train)\n",
    "\n",
    "# get R-squared score\n",
    "ols.score(features_test, target_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Discussion\n",
    "DummyRegressor allows us to create a very simple model that we can use as abaseline to compare against our actual model. This can often be useful to simulate a \"naive\" existing prediction process in a product or system. For example, a product might have been originally hardcoded to assume that all new users will spend $100 in the first month, regardless of their features. If we encode that assumption into a baseline model, we are able to concretely state the benefits of using a machine learning approach.\n",
    "\n",
    "DummyRegressor uses the strategy parameter to set the method of making predictions, including the mean or median value in the training set. Furthermore, if we set strategy to constant and use the constant parameter, we can set the dummy regressor to predict some constant value for every observation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.06510502029325727"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# create dummy regressor that predicts 20's for everything\n",
    "clf = DummyRegressor(strategy='constant', constant=20)\n",
    "clf.fit(features_train, target_train)\n",
    "\n",
    "# evaluate score\n",
    "clf.score(features_test, target_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One small note regarding score. By default, score returns the coefficient of determination (R-squared, $R^2$) score:\n",
    "\n",
    "$$R^2 = 1 - \\frac{\\sum(y_i - \\hat y_i)^2}{\\sum(y_i - \\bar y)^2}$$\n",
    "\n",
    "where $y_i$ is the true value of the target observation, $\\hat y_i$ is the predicted value, and $\\bar y$ is the mean value for the target vector\n",
    "\n",
    "The closer $R^2$ is to 1, the more of the variance in the target vector that is explained by the features.\n",
    "\n",
    "### 11.3 Creating a Baseline Classification Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Problem\n",
    "You want a simple baseline classifier to compare against your model\n",
    "\n",
    "#### Solution\n",
    "Use scikit-learn's DummyClassifer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.42105263157894735"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# load libraries\n",
    "from sklearn.datasets import load_iris\n",
    "from sklearn.dummy import DummyClassifier\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# load data\n",
    "iris = load_iris()\n",
    "\n",
    "# create target vector and feature matrix\n",
    "features, target = iris.data, iris.target\n",
    "\n",
    "# split into training and test set\n",
    "features_train, features_test, target_train, target_test = train_test_split(features, target, random_state=0)\n",
    "\n",
    "# create dummy classifier\n",
    "dummy = DummyClassifier(strategy='uniform', random_state=1)\n",
    "\n",
    "# \"train\" model\n",
    "dummy.fit(features_train, target_train)\n",
    "\n",
    "# get accuracy score\n",
    "dummy.score(features_test, target_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Discussion\n",
    "#### See Also\n",
    "scikit-learn documentation: DummyClassifier (http://scikit-learn.org/stable/modules/generated/sklearn.dummy.DummyClassifier.html)\n",
    "\n",
    "### 11.4 Evaluating Binary Classifier Predictions\n",
    "#### Problem\n",
    "Given a trained classification model, you want to evaluate it's quality\n",
    "\n",
    "#### Solution\n",
    "Use scikit-learn's cross_val_score to conduct cross-validation while using the `scoring` parameter to define one of a number of performance metrics, including accuracy, precision, recall, and $F_1$\n",
    "\n",
    "Accuracy is a common performance metric. It is simply the proportion of observations predicted corrected\n",
    "\n",
    "$$Accuracy = \\frac{TP + TN}{TP + TN + FP + FN}$$\n",
    "\n",
    "where:\n",
    "* $TP$ is the number of true positives. Observations that are part of the positive class (has the disease, pruchased the product, etc) and that we predicted correctly.\n",
    "\n",
    "* $TN$ is the number of true negatives. Observations that are part of the negative class (does not have the disease, did not purchase hte product, etc.) and that we predicted correctly.\n",
    "\n",
    "* $FP$ is the number of false positives. Also called Type 1 error. Observations predicted to be part of hte positive class that are actually part of the negative class.\n",
    "\n",
    "* $FN$ is the number of false negatives. Also called Type 2 error. Observations predicted to be part of the negative class that are actually part of the positive class.\n",
    "\n",
    "We can measure accuracy in three-fold (the default number of folds) cross-validation by setting `scoring='accuracy'`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.95170966, 0.9580084 , 0.95558223])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# load libraries\n",
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.datasets import make_classification\n",
    "\n",
    "# generate features matrix and target vector\n",
    "X, y = make_classification(n_samples = 10000,\n",
    "                           n_features = 3,\n",
    "                           n_informative = 3,\n",
    "                           n_redundant = 0,\n",
    "                           n_classes = 2,\n",
    "                           random_state = 1)\n",
    "\n",
    "# create logistic regression\n",
    "logit = LogisticRegression()\n",
    "\n",
    "# cross-validate model using accuracy\n",
    "cross_val_score(logit, X, y, scoring='accuracy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Precision is the proportion of every observation predicted to be positive that is actually positive. We can think about it as a measurement of noise in our predictions-- that is, when we predict something is positive, how likely we are to be right. Models with high precision are pessimistic in that they only predict an observation is of the positive class when they are very certain about it. Formally, precision is:\n",
    "\n",
    "$$Precision = \\frac{TP}{TP + FP}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.95252404, 0.96583282, 0.95558223])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# cross-validate model using precision\n",
    "cross_val_score(logit, X, y, scoring='precision')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall is the proportion of every positive observation that is truly positive. Recall measures the model's ability to identify an observation of the positive class. Models with high recall are optimistic in that they have a low bar for predicting that an observation is in the positive class:\n",
    "\n",
    "$$ Recall = \\frac{TP}{TP + FN}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.95080984, 0.94961008, 0.95558223])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# cross-validate model using recall\n",
    "cross_val_score(logit, X, y, scoring='recall')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The $F_1$ score is the harmonic mean (a kind of average used for ratios):\n",
    "$$ F_1 = 2 * \\frac{Precision * Recall}{Precision + Recall}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Discussion\n",
    "#### See Also\n",
    "* Accuracy paradox (https://en.wikipedia.org/wiki/Accuracy_paradox)\n",
    "\n",
    "### 11.5 Evaluating Binary Classifier Thresholds\n",
    "#### Problem\n",
    "You want to evaluate a binary classifier and various probability thresholds\n",
    "\n",
    "#### Solution\n",
    "The Recieving Operating Characteristic (ROC) curve is a common method for evaluating the the quality of a binary classifier. ROC compares the presence of true positives and false positives at every probability threshold (i.e. the probability at which an observatio nis predicted to be a class). By plotting the ROC curve, we can see how the model performs. A classifier that predicts every observation correctly woudl look like the solid light gray line in the following chart, going straight up to the top immediately. A classifier that predicts at random will appear as the diagonal line. The better the model, the closer it is to the solid line. In scikit-learn, we can use `roc_curve` to calculate the true and false positives at each threshold, then plot them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3XecFPX9x/HXB5AmTQSjUgQBC3Y9wS4GUayosWCLNSb+LLEl1tiiiVFjixV7FzUaEVHsXRBsoNgoKqcovRfh7vP74zt3LuuVveNmZ8v7+Xjsg92Z2ZnP7C3z2fl+Zz5fc3dEREQAGiUdgIiI5A4lBRERqaSkICIilZQURESkkpKCiIhUUlIQEZFKSgqyEjN73syOSTqOfGFmC81s/QS2283M3MyaZHvbcTCzz8ysXz3ed6SZvRhDSEVLSSFPmdk3ZrYkOij9aGb3mVmrVV2vu+/l7vc3RIyrysx2MLNXzWyBmc0zs2fNrHeC8bxuZiemTnP3Vu4+OabtbWBmT5jZzGj/x5nZWWbWOI7t1VeUnHquyjrcfRN3f72W7fwqEbr7w+6+x6psW1ampJDf9nP3VsCWwFbA+QnHUy9V/do1s+2BF4FngHWB7sAnwDtx/DLPtV/cZtYDGA1MBTZz97bAIUAJ0LqBt5XYvufa5y6Au+uRhw/gG2D3lNdXA8+lvG4GXAt8B/wE3A60SJk/CPgYmA9MAgZG018HTkxZ7njgc2AOMBJYL5p+O3BtWkzPAGdFz9cF/gvMAKYAp6csdynwJPBQtP0Tq9i/t4Bbq5j+PPBA9LwfUApcAMyMPpMjM/kMUt57LvAj8CCwBjA8inlO9LxztPyVQBmwFFgI3BxNd6Bn9Pw+4BbgOWAB4aDeIyWePYAvgXnArcAbVe17tOxDqX/PKuZ3i7Z9TLR/M4ELU+b3Ad4D5gLTgJuBpinzHTgF+BqYEk27kZCE5gMfADunLN84+pwnRfv2AdAFeDNa16LoczksWn5fwvdrLvAusHnad/dcYBywDGhCyvc5in1sFMdPwHXR9O+ibS2MHtsDxwJvp6x7E+AlYHb03guS/r+ab4/EA9Cjnn+4lf8TdQbGAzemzL8BGAa0J/yyfBb4ZzSvT3RgGkA4W+wEbBTNe73iQAUcAEwENo7+414EvBvN2yU6gFj0eg1gCSEZNIoOGhcDTYH1gcnAntGylwLLo/U3IiVZRfNbEg7Au1Wx38cB06Ln/YAVwHWEBLBrdHDaMIPPoOK9/4re2wJYE/hdtP3WwBPA/1K2XfnZpExLTwqzo8+3CfAw8Fg0r0N0kDsomvfn6DOoLin8CBxXw9+/W7TtO6PYtyAcYDeO5m8DbBdtqxshsZ+RFvdL0WdTkSiPij6DJsDZUQzNo3l/IXzHNgQs2t6a6Z9B9HprYDrQl5BMjiF8X5ulfHc/JiSVFinTKr7P7wFHR89bAdul7XOTlG0dS5QUor/ZtCj25tHrvkn/X823R+IB6FHPP1z4T7SQ8KvNgVeAdtE8IxwcU3+lbs8vvwjvAK6vZr2VBz7Cr/ITUuY1AhYD60Xb+A7YJZr3B+DV6Hlf4Lu09Z4P3Bs9vxR4s4Z96xzt00ZVzBsILI+e9yMc2FdPmf848LcMPoN+wM8VB71q4tgSmFPVZ5MyLT0p3JUyb2/gi+j574H3UuYZIalWlxSWE529VTO/4gDZOWXa+8DgapY/A3g6Le7f1vIdmwNsET3/EhhUzXLpSeE24O9py3wJ7Jry3T2+iu9zRVJ4E7gM6FDNPleXFA4HPor7/16hP9SnkN8OcPfWhAPcRoRfowAdCb92PzCzuWY2F3ghmg7hF9qkDNa/HnBjyjpmEw5mnTz8L3yM8B8R4AjCL+OK961b8b7ovRcAv0lZ99QatjsHKAfWqWLeOoSmkspl3X1RyutvCWcrtX0GADPcfWnFCzNraWZ3mNm3ZjafcHBqV8eO3R9Tni8m/NIliqlyn6PPr7SG9cyi6v3PaHtRJ/Xw6CKE+cA/+OX7UWGlv4GZnW1mn0ed2nOBtinvyfQ7A+Hvf3ba378L4TOocttpTgA2AL4wszFmtm+G261LjFINJYUC4O5vEH6lXhtNmkloytnE3dtFj7YeOqUh/IfskcGqpwJ/TFlHO3dv4e7vRvMfBQ42s/UIZwf/TXnflLT3tXb3vVPDrmF/FhGaEA6pYvahhLOiCmuY2eopr7sCP2TwGVQVw9mE5pG+7t6G0EQGIRHWGHMGphHOgMIKzSz1dRVeJjRl1ddtwBdAr2hfLuCX/ahQuT9mtjOhnf9QYA13b0doYqx4T6bfmYplr0z7+7d090er2nY6d//a3Q8H1iI07z0Z/Y1r+/zrEqNUQ0mhcNwADDCzLd29nNDWfL2ZrQVgZp3MbM9o2buB48ysv5k1iuZtVMU6bwfON7NNonW0NbPKA7W7f0TolL0LGOnuc6NZ7wPzzexcM2thZo3NbFMz27YO+3MecIyZnW5mrc1sDTO7gtAEdFnaspeZWdPowLYv8EQGn0FVWhMSyVwzaw9ckjb/J0L/SH08B2xmZgdEV9ycAqxdw/KXADuY2TVmtnYUf08ze8jM2mWwvdaEPoyF0d/25AyWX0H4ezYxs4uBNinz7wL+bma9LNjczNaM5qV/LncCfzKzvtGyq5vZPmaW0VVTZnaUmXWM/oYV36myKLZyqv8bDAfWNrMzzKxZ9L3pm8k25RdKCgXC3WcADxDa0yH86psIjIqaD14m/ArG3d8ndNheT/g1+AbhlD99nU8Tfqk9Fq3jU2CvtMUeBXYHHkl5XxmwH6FNfgrhV/tdhOaITPfnbWBPQsfsNEKz0FbATu7+dcqiPxKam34gNF/9yd2/qO0zqMYNhE7bmcAoQnNTqhsJZ0ZzzOymTPcl2p+ZhDOfqwlNQ70JV9gsq2b5SYQE2A34zMzmEc7ExhL6kWpzDqFJbwHhID20luVHEvqQviJ81ktZuYnnOkJ/zYuEZHM34bOC0Ed0f9RUdKi7jyX0Md1M+NtMJLT9Z2ogYZ8XEj7zwe6+1N0XE64Ceyfa1napb3L3BYSLJ/YjfC++Bnarw3aFX64cEck7Fu6Afcjda2qGyUlm1ojQp3Cku7+WdDwiFXSmIJIlZranmbUzs2b80sY/KuGwRFaipCCSPdsTro6ZSWjiOMDdlyQbksjK1HwkIiKVdKYgIiKV8q4YVYcOHbxbt25JhyEiklc++OCDme7esbbl8i4pdOvWjbFjxyYdhohIXjGzbzNZTs1HIiJSSUlBREQqKSmIiEglJQUREamkpCAiIpViSwpmdo+ZTTezT6uZb2Z2k5lNtDAg+dZxxSIiIpmJ80zhPkK1w+rsBfSKHicR6r+LiEiCYrtPwd3fNLNuNSwyiDAAuxNKG7czs3XcfVoc8bz55pv8/PPPtGzZMo7Vi0iemb5gGTMXVlm5POc08nKasILlzdpxwn671P6GVZDkzWudWLlee2k07VdJwcxOIpxN0LVr13ptbNmyZZSVldXrvSLFLp8OoJlasHQFAK2b5/Y9vKuXL2SdFd9Tbo2Y0izjIUnqLclPI31oQKhmuD13HwIMASgpKalXBb/VVw8jNu6www71ebtIXnlk9Hc88/H3Dba+0VMWA9C3e/sGW2cuGLRlJ47oW78fmrFbMhde+ht8+AC0Xx/2/w902yn2zSaZFEoJA21X6EwYPUtE6ig9CYyeMhtouIN43+7tc/sAWmjKy+DuPWDW17Djn6Hf+bBai9rf1wCSTArDgFPN7DHCoO/z4upPEMl1q/rLPj0J6CCepxbPhhZrQKPG0P9v0KYTdMruhZmxJQUzexToB3Qws1LCQOSrAbj77cAIYG/C+K2LCWMGixScTA74q/rLXkkgz7nDuMfhhXNh90thm2Nh4/0SCSXOq48Or2W+A6fEtX2RbKntoJ/JAV8H9SI2rxSGnwlfvwidt4Uu2yUaTm53u4vkkOoO/rUd9HXAl2qNfxKePQO8DAZeBX1OCk1HCVJSkKJV13b86g7+OuhLvTVvB523gf1uhDW6JR0NoKQgBSjTg31d2/F18JdVVrYCRt0CZT/DLn+BXrtDz/5gVV2hnwwlBSkIqYkg04O9DvKSVT+Oh2dOhWkfwyYHhs5ls5xKCKCkIHmkpjOA1ESgg73klBXL4M1r4O3rw+Wmh9wPvQflXDKooKQgOa8iGdR0BqBEIDlr1iR4+wbY7BDY8x/QMrfvCldSkJyRydU9OvBLXli2EL4cAZsfCr/pDaeOgfbdk44qI0oKkphMSzMoGUhemfQqPPtnmDsV1tkCOm6YNwkBlBQkZpn2A1T8q4O/5K0lc+DFi+Cjh2DNnnDciJAQ8oySgjSouhRmUxKQglFeBnfvCbMmwk5nwa7nwmrNk46qXpQUZJXUlgR04JeCtmhWSgG7i6FtZ1h3y6SjWiVKClJnNd0ToCQgRcEdPnkMXjgvFLArOQ423jfpqBqEkoLUySOjv+OCp8cDuidAitTc70K9okmvQJe+sN6OSUfUoJQUpEbVNQ/948DNlAik+HwyFJ47K5wp7HUNbHsiNGqUdFQNSklBqlTdDWM6M5Citvqa4exgvxugXWH+H1BSkCovG9UNYyJA2XJ49z9QvgJ2/Sv03B165FYBu4ampFCkaisgp2QgRW/aJ6GA3Y/jYNPf5WwBu4ampFBEqksESgAiKZYvhTf+Be/cCC3XhEMfhN77Jx1V1igpFIGq+geUCESqMXtyaDLa4nDY84pwH0IRUVIocOmXkCoRiFRh2UL4YjhsMTgUsDttbM6MhJZtSgoFLDUh6BJSkWpMfDncdzCvFNbdKtQrKtKEAEoKBSm9uUgJQaQKi2fDyAvgk0ehwwZw/At5WcCuoSkpFBg1F4lkoLwM7t4j9B/sfE4YLzlPC9g1NCWFAlDVVUU6OxCpwqKZ0KJ9KGA34DJo2wXW2TzpqHJKYd2fXYQqzgxSryxSQhBJ4x7GOfjP1vDhfWHaRvsoIVRBZwp5TB3JIhmY820YCW3ya9B1B+i2S9IR5TQlhTylhCCSgU8eg+FnhbuQ9/k3bHN8wRWwa2hKCnlICUEkQ6t3hPV2gH2vh3Zdko4mLygp5BklBJEalC2Hd26A8nLody707B8ekjElhTyihCBSgx8+DgXsfhoPmx3ySwE7qRMlhTyhhCBSjeVL4PWrQr2i1TvAYQ8XzNCYSYi1x8XMBprZl2Y20czOq2J+VzN7zcw+MrNxZrZ3nPHkKyUEkRrM+QbeuwW2PAJOGa2EsIpiO1Mws8bALcAAoBQYY2bD3H1CymIXAY+7+21m1hsYAXSLK6Z8VXFjmhKCSGTpfPj8WdjqSFhrYzj9w4IdCS3b4mw+6gNMdPfJAGb2GDAISE0KDrSJnrcFfogxnrzWt3t7JQQRgK9ehOFnwoIfoHNJqFekhNBg4mw+6gRMTXldGk1LdSlwlJmVEs4STqtqRWZ2kpmNNbOxM2bMiCPWnPXI6O8q71YWKWqLZsFTJ8Ejh0CzVnD8iypgF4M4zxSq6vb3tNeHA/e5+7/NbHvgQTPb1N3LV3qT+xBgCEBJSUn6OgpSeqXTQVum51ORIlJeBvfsEfoPdj0Xdj4bmjRLOqqCFGdSKAVS7xbpzK+bh04ABgK4+3tm1hzoAEyPMa6cVtUoaap0KkVr4XRo2SEUsNvjilDAbu1Nk46qoMWZFMYAvcysO/A9MBg4Im2Z74D+wH1mtjHQHCiu9qE0z3z8PROmzVcykOLmDh89CCMvgt0vgW1PgA33SjqqohBbUnD3FWZ2KjASaAzc4+6fmdnlwFh3HwacDdxpZmcSmpaOdfeiaB6qSe912jD0j9snHYZIMmZPgWdPhylvwno7wfr9ko6oqMR685q7jyB0IKdOuzjl+QRgxzhjyCcVncp9u7dPOhSRZHz8CDx3NljjUK9o62NVwC7LdEdzDlCnskik9drQfRfY5zpoq/8HSVBSSJiGz5SituJnePt68HLY7Xzo8dvwkMQoKSRMdytL0fr+g1DAbvoE2HywCtjlCCWFBKX2ISghSNH4eTG8diWMuhVarQ2HP6Yri3KIkkKCKs4S1IcgRWXut/D+ENj6GBhwGTRvm3REkkJJISE6S5CisnReVMDuqKiA3UfQtnPSUUkVlBQSkNq5rLMEKXhfjYRnz4CFP0LnPtBxAyWEHKakkEXpl56qc1kK2qKZ8MJ5MP4JWKs3HPZQSAiS05QUskSXnkpRKS+De/aEOd9CvwtgpzOhSdOko5IMZJQUzKwp0NXdJ8YcT8HSpadSFBb8BKt3jArYXRnGOfhN76SjkjqoNSmY2T7AdUBToLuZbQlc4u4Hxh1cvqtoLgIqi9wpIUhBKi+HD++DFy+GAZfCtifChgOTjkrqIZOiIpcDfYG5AO7+MdAzzqAKRUXFUwhF7tSpLAVp1iR4YP8wGlqnraBH/6QjklWQSfPRcnefayvfaVj0lUwzpYqnUtA+eigUsGvcFPa7Cbb+ve5KznOZJIXPzexQoFE0NsKfgVHxhiUieaFt53BmsM+10GbdpKORBpBJ89GpwDZAOfAUsJSQGESk2KxYBq/9E169Mrxevx8c/ogSQgHJJCns6e7nuvtW0eM8QIVKalFxx7JIwSgdC3fsCm9cBfNKQwE7KTiZJIWLqph2YUMHUmhU10gKxs+L4IUL4K7dYdl8OOJxOPA29R0UqGr7FMxsT2Ag0MnMrkuZ1YbQlCTVUF0jKShzp8KYu6DkeNj9UmjeJumIJEY1dTRPBz4l9CF8ljJ9AXBenEHlO50lSN5bMhcmPAPbHANrbRQVsNP3uRhUmxTc/SPgIzN72N2XZjGmgqCzBMlbXzwHw8+CRTOg6/ZRATslhGKRySWpnczsSqA30LxiorurspVIIVk4A57/K3z2FPxmUzj8URWwK0KZdDTfB9wLGOGqo8eBx2KMKa/pqiPJS+VlcM8e8MVw+O1FcNLr0GnrpKOSBGRyptDS3Uea2bXuPgm4yMzeijuwfKRxEiTvzJ8GrX4TCtgN/FcoYLfWRklHJQnK5ExhmYUaF5PM7E9mth+wVsxx5SVVQpW8UV4erii6eVsYe3eYtsEeSgiS0ZnCmUAr4HTgSqAtcHycQeWbimqoqoQqeWHmRHj2dPj2nXBHcq8BSUckOaTWpODuo6OnC4CjAcxMY+lFqho8RyRnffgAjPgLNGkGg26BLY/UTWiykhqTgpltC3QC3nb3mWa2CXAu8Fug6BNDakJQk5HkhXZdoefusM+/ofXaSUcjOajaPgUz+yfwMHAk8IKZXQi8BnwCFP11akoIkhdWLINX/h4eEJqLBj+shCDVqulMYRCwhbsvMbP2wA/R6y+zE1ruUkKQvPDdaBh2Ksz8CrY6KhSwU1OR1KKmpLDU3ZcAuPtsM/tCCUEJQfLAsoXw6t9h9B1hvIOj/huajEQyUFNSWN/MnoqeG9At5TXuflBtKzezgcCNQGPgLne/qoplDgUuJYzm9om7H5F5+Nmny04l580rhbH3Qp8/QP+LoVnrpCOSPFJTUvhd2uub67JiM2sM3AIMAEqBMWY2zN0npCzTCzgf2NHd55hZXtz/oMtOJecsmQOf/Q9Kjgv3Gvz5E2izTtJRSR6qqSDeK6u47j7ARHefDGBmjxH6KSakLPMH4BZ3nxNtc/oqblOk+Hz+bBgnedFM6LYTdOilhCD1lskdzfXVCZia8ro0mpZqA2ADM3vHzEZFzU2/YmYnmdlYMxs7Y8aMmMKtneoaSU5Z8BM8/nsYehS0Wgv+8GpICCKrIJM7muurqssc0sfvawL0AvoR7nt4y8w2dfe5K73JfQgwBKCkpCSxMQA1ToLkjPIyuHcgzPs+9BvscDo0Xi3pqKQAZJwUzKyZuy+rw7pLgS4przsTLmtNX2aUuy8HppjZl4QkMaYO28kq9SdIouZ9D63XCQXs9roa2q2n8tbSoGptPjKzPmY2Hvg6er2Fmf0ng3WPAXqZWXczawoMBoalLfM/YLdovR0IzUmT6xC/SHEoLw+XmKYWsOs1QAlBGlwmfQo3AfsCswDc/ROiA3lN3H0FcCowEvgceNzdPzOzy81s/2ixkcAsM5tAuFv6L+4+q+67ET/1J0hiZnwF9+4VBsDpuh1ssGfSEUkBy6T5qJG7f2sr3wlZlsnK3X0EMCJt2sUpzx04K3rkNPUnSCI+uD8UsFutBRxwO2wxWHclS6wySQpTzawP4NG9B6cBX8UbVm5Sf4JkXfvusOFA2PvacIWRSMwySQonE5qQugI/AS9H04pGRdNR3+7tkw5FCt3ypfDGv8Lz3S+B7ruEh0iWZJIUVrj74NgjyWFqOpKs+G4UPHMqzPoatv69CthJIjJJCmOiS0WHAk+5+4KYY8pJajqS2CxbAK9cDu/fCe26wFFPQc/+SUclRarWq4/cvQdwBbANMN7M/mdmRX3mINKg5v8QRkTr+0c4+T0lBElURmUu3P1ddz8d2BqYTxh8pyjoUlSJxeLZMOau8LzjhqGA3V7/gmatko1Lil6tzUdm1opQyG4wsDHwDLBDzHHlDPUnSINyhwnPwIhzQmXT7ruGekUaCU1yRCZ9Cp8CzwJXu/tbMceTk9SfIA1iwY+hmukXw2GdLeHop1XATnJOJklhfXcvjz0SkUJWXgb3DIQF02DA5bDdKdA4znqUIvVT7bfSzP7t7mcD/zWzX1UmzWTkNZGiN68UWq8bCtjtcy206wYdeiYdlUi1avqpMjT6t04jrhUS3bQm9VZeFi4xfeWycGbQ5w8aJ1nyQk0jr70fPd3Y3VdKDGZ2KrCqI7PlPHUyS73M+DLchFb6PvQcABtUOXaUSE7K5JLU46uYdkJDB5Kr1MksdTL2Xrh9J5g1EQ4cAkc+EW5IE8kTNfUpHEa4DLW7mT2VMqs1MLfqd4kUuTV7wEb7hgFwWnVMOhqROqupT+F9whgKnYFbUqYvAD6KMyiRvLF8Cbz+T8BgwGUqYCd5r6Y+hSnAFEJVVBFJ9807MOw0mD0JSo5XATspCDU1H73h7rua2Rwg9ZJUI4yPo0typDgtnQ8vXxqGxVyjG/x+GKy/a9JRiTSImpqPKobc7JCNQETyxoIf4eNHYPtTYbcLoOnqSUck0mCqvfoo5S7mLkBjdy8Dtgf+COh/gRSXRbPCfQcAHTeAM8bBnlcqIUjByeSS1P8RhuLsATxAKIr3SKxR5QBVRxUg9BN8+l+4pQ+8cD7MnBima2hMKVCZFF8pd/flZnYQcIO732RmBX310SOjv+OCp8cDunGtqM2fBs+dBV+OgHW3gkHDVKJCCl5Gw3Ga2SHA0cAB0bTV4gspeRV3Mv/jwM1041qxKi+De/cKBez2uAL6nqwCdlIUMvmWHw/8H6F09mQz6w48Gm9YydOdzEVq7nfQplNUwO7f4eqiNXskHZVI1mQyHOenwOnAWDPbCJjq7lfGHllC1JdQpMrL4N2b4eY+MObuMK1nfyUEKTqZjLy2M/Ag8D3hHoW1zexod38n7uCSoCJ4ReinCTDsVPj+g1C8bqN9ko5IJDGZNB9dD+zt7hMAzGxjQpIoiTOwJKnpqIiMuRuePxeat4Hf3Q2b/k53JUtRyyQpNK1ICADu/rmZNY0xJpH4VZSk6LghbHIADLwKVtd9miKZJIUPzewOwtkBwJEUYEG8R0Z/xzMff8+EafPpvU6bpMORuPy8GF67MnQkD7gcuu0UHiICZHbz2p+AScBfgXOByYS7mgtKakJQf0KBmvIW3LYDvHcz/LwonC2IyEpqPFMws82AHsDT7n51dkJKTu912jD0j9snHYY0tKXz4KWL4YP7YI3ucMyzKm8tUo1qzxTM7AJCiYsjgZfMrKoR2ERy34KfYNzjsMNpcPK7SggiNaip+ehIYHN3PwTYFji5ris3s4Fm9qWZTTSz82pY7mAzczNL5Iom3ZtQgBbNhNF3hOcdN4Azxoc7k5u2TDYukRxXU/PRMndfBODuM8wsk/6HSmbWmDBi2wCgFBhjZsNSr2SKlmtNuDludJ0ib0C6N6GAuMP4J+H5v8KyBdCjf6hXpCuLRDJSU1JYP2VsZgN6pI7V7O4H1bLuPsBEd58MYGaPAYOACWnL/R24GjinLoE3NN2bUADmlcLws+DrkdCpBAbdrAJ2InVUU1L4Xdrrm+u47k7A1JTXpUDf1AXMbCugi7sPN7Nqk4KZnQScBNC1a8MeuCuajvp210Byea1sBdy3DyycDnv+E/r+MVx2KiJ1UtMYza+s4rqrui208hrAqDnqeuDY2lbk7kOAIQAlJSUNeh2hmo7y3JxvoW3nUMF03xtCAbv23ZOOSiRv1amfoI5KCaO2VegM/JDyujWwKfC6mX0DbAcMy2Znc+pZgpqO8kzZCnjnpjD4zZi7wrQeuykhiKyiOAvEjwF6RaW2vwcGA0dUzHT3eaSM/2xmrwPnuPvYGGNaic4S8tSPn4YCdj98BBvuAxvvn3REIgUj46RgZs3cfVmmy7v7CjM7FRgJNAbucffPzOxyYKy7D6t7uA1PZwl55v074YXzoHk7OPhe2ORAFbATaUCZlM7uA9wNtAW6mtkWwInuflpt73X3EcCItGkXV7Nsv0wCliJVUcBurd6hkume/4TV10w6KpGCk8mZwk3AvoS7m3H3T8xst1ijEqnw8yJ49YpwJdEeV0C3HcNDRGKRSUdzI3f/Nm1aWRzBiKxk8utw6/Yw6lZY8bMK2IlkQSZnClOjJiSP7lI+Dfgq3rCkqC2ZCy9eBB89CO17wHHPw3o7JB2VSFHIJCmcTGhC6gr8BLxMPeogiWRs0Qz49CnY8Qzodx6s1iLpiESKRq1Jwd2nEy4nFYnPwunw6X9hu5OhQ69QwE4dySJZl8nVR3eScidyBXc/KZaIpLi4h7LWL5wbOpV77QFr9lBCEElIJs1HL6c8bw4cyMo1jUTqZ+5UGH4mTHwJOvcJBezW7JF0VCJFLZPmo6Gpr83sQeCl2CKS4lBRwG7RTNjratj2RBWwE8kB9Slz0R1Yr6EDyTZVR03I7CnQrmsoYLf/TWF4zDXy/uskUjBqvU/BzOaY2ezoMZdwlnBB/KHFS3WPsqxsBbx9PdzSN5SqAFi/nxKCSI6p8UzBzAzYglDQDqDcvXDuIFLdoyyZNi4UsJvIzRI6AAAQEUlEQVT2CWy0L2xyQNIRiUg1akwK7u5m9rS7b5OtgKTAjB4CI8+HFu3h0Aeg96CkIxKRGmTSp/C+mW3t7h/GHo0UjooCdr/ZBDY7FPa8Elqq/0Yk11WbFMysibuvAHYC/mBmk4BFhBHV3N23zlKMkk+WLYRX/w6NmoREoAJ2InmlpjOF94GtATUAS2YmvgLPngHzpoYxkivOFkQkb9SUFAzA3SdlKRbJV0vmwMgL4eOHYc1eUQG77ZOOSkTqoaak0NHMzqpuprtfF0M8WaF7FBrYopkw4RnY6SzY9VxYrXnSEYlIPdWUFBoDrYjOGAqJ7lFoAAt+gk+fhO1P+aWAnTqSRfJeTUlhmrtfnrVIskz3KNSTO3zyKLxwPixfAhsMDPWKlBBECkKtfQoileZ8C8PPgEmvQpftYP//qICdSIGpKSn0z1oUkvvKVsD9+8Li2bD3tVByAjTKZDRXEckn1SYFd5+dzUAkR82aBGt0CwXsBt0SnrdTs5tIodJPPala2XJ481q4dbtfCth130UJQaTA1ad0thS6Hz4OBex+HA+9D4BND0o6IhHJEiUFWdmo22HkBbB6BzjsIdh4v6QjEpEsUlKQoKIkxTqbwxaHw55XQIs1ko5KRLJMSaHYLVsAL18GTZqFAnbr7RAeIlKUiq6juaLEhQBfvwy3bg9j7gpnCoUzfpKI1FPRnSmoxAXhXoORF4Q7kztsCCe8CF36JB2ViOSAoksKoBIXLJ4Nnw+HXf4Ku5wTmo5ERIi5+cjMBprZl2Y20czOq2L+WWY2wczGmdkrZqZR3OOy4Ed456bQRNShJ5w5Hn57oRKCiKwktqRgZo2BW4C9gN7A4WbWO22xj4ASd98ceBK4Oq54ipY7fPgg3NwHXrsSZk8O03VlkYhUIc4zhT7ARHef7O4/A48BK43a7u6vufvi6OUooHOM8RSfOd/AgweEG9HW3hT+9I4K2IlIjeLsU+gETE15XQr0rWH5E4Dnq5phZicBJwF07VrEfQF1UbYC7t8PFs+Bfa6DbY5TATsRqVWcSaGq0ttVXvNoZkcBJcCuVc139yHAEICSkhJdN1mTlQrY3Qrtu0NbnYCJSGbi/OlYCnRJed0Z+CF9ITPbHbgQ2N/dl8UYT2ErWw5vXBMVsBsSpnXfWQlBROokzjOFMUAvM+sOfA8MBo5IXcDMtgLuAAa6+/QYYyls338Iw06Dnz6FTX8Hmx6cdEQikqdiO1Nw9xXAqcBI4HPgcXf/zMwuN7P9o8WuIYwD/YSZfWxmw+KKB2D6gmWFdzfzqNvgrv6weBYMfhQOvgdadUw6KhHJU7HevObuI4ARadMuTnm+e5zbTzdzYWidKoi7mSsK2K27FWx1NAy4HFq0SzoqEclzRXdHc97fzbx0Prx8CTRpDgP/CV23Cw8RkQagaxTzyVcvho7kD+6DRo1VwE5EGlzRnSnkpUWz4IXzYPzj0HFjOPQB6FySdFQiUoCUFPLB0rnw1Quw63mw89nQpGnSEYlIgVJSyFXzf4Bxj8OOfw6lKc4Yr45kEYmdkkKucYcP74cX/xZuSNt4v5AUlBBEJAuUFHLJ7Mkw7HT45i3otjPsd6MK2IlIVikp5IqyFXD/IFgyB/a9AbY+RgXsRCTrlBSSNvNrWKN7KGB34G3hedsCuLlORPKSfoomZcXP8PpVcOv2MObOMK3bTkoIIpIonSkkofSDMPDN9Amw2SGw2aFJRyQiAigpZN97t8KLF0KrteHwobDhwKQjEhGppKSQLRUF7DptEzqRB1wGzdsmHZWIyEqUFOK2dB68dDE0aQF7XQVd+4aHiEgOUkdznL58Hm7pCx8+EEpTqICdiOQ4nSnEYdFMeP5c+PRJWGsTGPxwaDYSEclxSgpxWDoPvn4J+l0AO52pAnYikjeUFBrKvFIYNxR2OiuUpjhzvDqSRSTvKCmsqvJy+OBeeOkS8DLofUBICkoIIpKHiiYpTF+wjAVLVzTsSmdNCgXsvn0buu8aCti1796w2xARyaKiSQozFy4DYNCWDVRGomwFPHBA6D/Y/2bY6qhwH4KISB4rmqQA0Lp5E47o23XVVjLjS2jfIxSwO+iOUMCuzToNE6CISMJ0n0KmViyD1/4Bt+0A7w8J09bbQQlBRApKUZ0p1NvUMaGA3YwvYPPBsMXgpCMSEYmFkkJt3v1PGBqzTSc48knoNSDpiEREYqOkUJ3y8jDyWec+UHI87H4pNG+TdFQiIrFSUki3ZG4obb1aS9j7GhWwE5Gioo7mVJ8PDwXsPn4UmrZSATsRKTo6UwBYOANGnAMT/gdrbwZHDIV1t0w6KhGRrFNSAFg2Hya/Br/9G+z4Z2i8WtIRiYgkoniTwtypMO4x2PmcqIDdZ9CsddJRiYgkKtY+BTMbaGZfmtlEMzuvivnNzGxoNH+0mXWLMx4gXFX0/p1w63bw1nUwe3KYroQgIhJfUjCzxsAtwF5Ab+BwM+udttgJwBx37wlcD/wrrngAmvkyuG+f0H/QeVv4v1HhLEFERIB4zxT6ABPdfbK7/ww8BgxKW2YQcH/0/Emgv1k8VeUMp+vyKTD9Mxh0Kxz9NKyxXhybEhHJW3H2KXQCpqa8LgXSL/ivXMbdV5jZPGBNYGbqQmZ2EnASQNeu9Sxo13R1frIucMr70Hrt+q1DRKTAxZkUqvrFn37hfybL4O5DgCEAJSUl9bp54IT9dqnP20REikqczUelQJeU152BH6pbxsyaAG2B2THGJCIiNYgzKYwBeplZdzNrCgwGhqUtMww4Jnp+MPCqu24jFhFJSmzNR1EfwanASKAxcI+7f2ZmlwNj3X0YcDfwoJlNJJwhqCa1iEiCYr15zd1HACPSpl2c8nwpcEicMYiISOZUEE9ERCopKYiISCUlBRERqaSkICIilSzfrgA1sxnAt/V8ewfS7pYuAtrn4qB9Lg6rss/ruXvH2hbKu6SwKsxsrLuXJB1HNmmfi4P2uThkY5/VfCQiIpWUFEREpFKxJYUhSQeQAO1zcdA+F4fY97mo+hRERKRmxXamICIiNVBSEBGRSgWZFMxsoJl9aWYTzey8KuY3M7Oh0fzRZtYt+1E2rAz2+Swzm2Bm48zsFTPL+7FIa9vnlOUONjM3s7y/fDGTfTazQ6O/9Wdm9ki2Y2xoGXy3u5rZa2b2UfT93juJOBuKmd1jZtPN7NNq5puZ3RR9HuPMbOsGDcDdC+pBKNM9CVgfaAp8AvROW+b/gNuj54OBoUnHnYV93g1oGT0/uRj2OVquNfAmMAooSTruLPydewEfAWtEr9dKOu4s7PMQ4OToeW/gm6TjXsV93gXYGvi0mvl7A88TRq7cDhjdkNsvxDOFPsBEd5/s7j8DjwGD0pYZBNwfPX8S6G9mVQ0Nmi9q3Wd3f83dF0cvRxFGwstnmfydAf4OXA0szWZwMclkn/8A3OLucwDcfXqWY2xomeyzA22i52359QiPecXd36TmESgHAQ94MApoZ2brNNT2CzEpdAKmprwujaZVuYy7rwDmAWtmJbp4ZLLPqU4g/NLIZ7Xus5ltBXRx9+HZDCxGmfydNwA2MLN3zGyUmQ3MWnTxyGSfLwWOMrNSwvgtp2UntMTU9f97ncQ6yE5CqvrFn37dbSbL5JOM98fMjgJKgF1jjSh+Ne6zmTUCrgeOzVZAWZDJ37kJoQmpH+Fs8C0z29Td58YcW1wy2efDgfvc/d9mtj1hNMdN3b08/vASEevxqxDPFEqBLimvO/Pr08nKZcysCeGUs6bTtVyXyT5jZrsDFwL7u/uyLMUWl9r2uTWwKfC6mX1DaHsdluedzZl+t59x9+XuPgX4kpAk8lUm+3wC8DiAu78HNCcUjitUGf1/r69CTApjgF5m1t3MmhI6koelLTMMOCZ6fjDwqkc9OHmq1n2OmlLuICSEfG9nhlr22d3nuXsHd+/m7t0I/Sj7u/vYZMJtEJl8t/9HuKgAM+tAaE6anNUoG1Ym+/wd0B/AzDYmJIUZWY0yu4YBv4+uQtoOmOfu0xpq5QXXfOTuK8zsVGAk4cqFe9z9MzO7HBjr7sOAuwmnmBMJZwiDk4t41WW4z9cArYAnoj7179x9/8SCXkUZ7nNByXCfRwJ7mNkEoAz4i7vPSi7qVZPhPp8N3GlmZxKaUY7N5x95ZvYoofmvQ9RPcgmwGoC7307oN9kbmAgsBo5r0O3n8WcnIiINrBCbj0REpJ6UFEREpJKSgoiIVFJSEBGRSkoKIiJSSUlBco6ZlZnZxymPbjUs2626apJ13ObrUSXOT6ISERvWYx1/MrPfR8+PNbN1U+bdZWa9GzjOMWa2ZQbvOcPMWq7qtqU4KClILlri7lumPL7J0naPdPctCMUSr6nrm939dnd/IHp5LLBuyrwT3X1Cg0T5S5y3klmcZwBKCpIRJQXJC9EZwVtm9mH02KGKZTYxs/ejs4txZtYrmn5UyvQ7zKxxLZt7E+gZvbd/VKd/fFTnvlk0/Sr7ZXyKa6Npl5rZOWZ2MKG+1MPRNltEv/BLzOxkM7s6JeZjzew/9YzzPVIKoZnZbWY21sI4CpdF004nJKfXzOy1aNoeZvZe9Dk+YWatatmOFBElBclFLVKajp6Opk0HBrj71sBhwE1VvO9PwI3uviXhoFwalT04DNgxml4GHFnL9vcDxptZc+A+4DB334xQAeBkM2sPHAhs4u6bA1ekvtndnwTGEn7Rb+nuS1JmPwkclPL6MGBoPeMcSChrUeFCdy8BNgd2NbPN3f0mQl2c3dx9t6j0xUXA7tFnORY4q5btSBEpuDIXUhCWRAfGVKsBN0dt6GWEmj7p3gMuNLPOwFPu/rWZ9Qe2AcZE5T1aEBJMVR42syXAN4TyyxsCU9z9q2j+/cApwM2E8RnuMrPngIxLc7v7DDObHNWs+TraxjvReusS5+qEsg+po24damYnEf5fr0MYcGZc2nu3i6a/E22nKeFzEwGUFCR/nAn8BGxBOMP91aA57v6ImY0G9gFGmtmJhDLD97v7+Rls48jUgnlmVuUYG1E9nj6EImyDgVOB39ZhX4YChwJfAE+7u1s4QmccJ2EEsquAW4CDzKw7cA6wrbvPMbP7CIXh0hnwkrsfXod4pYio+UjyRVtgWlQj/2jCr+SVmNn6wOSoyWQYoRnlFeBgM1srWqa9ZT4+9RdANzPrGb0+GngjaoNv6+4jCJ24VV0BtIBQvrsqTwEHEMYBGBpNq1Oc7r6c0Ay0XdT01AZYBMwzs98Ae1UTyyhgx4p9MrOWZlbVWZcUKSUFyRe3AseY2ShC09GiKpY5DPjUzD4GNiIMWTiBcPB80czGAS8RmlZq5e5LCRUonzCz8UA5cDvhADs8Wt8bhLOYdPcBt1d0NKetdw4wAVjP3d+PptU5zqiv4t/AOe7+CWFs5s+AewhNUhWGAM+b2WvuPoNwZdSj0XZGET4rEUBVUkVEJIXOFEREpJKSgoiIVFJSEBGRSkoKIiJSSUlBREQqKSmIiEglJQUREan0/3fBD1NvuWd1AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# load libraries\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.datasets import make_classification\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import roc_curve, roc_auc_score\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# create feature matrix and target vector\n",
    "features, target = make_classification(n_samples=10000,\n",
    "                                       n_features=10,\n",
    "                                       n_classes=2,\n",
    "                                       n_informative=3,\n",
    "                                       random_state=3)\n",
    "\n",
    "# split into training and test sets\n",
    "features_train, features_test, target_train, target_test = train_test_split(features, target, test_size=0.2, random_state=1)\n",
    "\n",
    "# create classifier\n",
    "logit = LogisticRegression()\n",
    "\n",
    "# train model\n",
    "logit.fit(features_train, target_train)\n",
    "\n",
    "# get predicted probabilities\n",
    "target_probabilities = logit.predict_proba(features_test)[:,1]\n",
    "\n",
    "# create true and positive rates\n",
    "false_positive_rate, true_positive_rate, threshold = roc_curve(target_test, target_probabilities)\n",
    "\n",
    "# plot ROC curve\n",
    "plt.title(\"Reciever Operating Characteristic\")\n",
    "plt.plot(false_positive_rate, true_positive_rate)\n",
    "plt.plot([0, 1], ls=\"--\")\n",
    "plt.plot([0, 0], [1,0], c=\".7\"), plt.plot([1, 1], c=\".7\")\n",
    "plt.ylabel(\"True Positive Rate\")\n",
    "plt.xlabel(\"False Positive Rate\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Discussion\n",
    "Up until now we have only examined models based on the values they predict. However, in amny leanring algorithms those predicted values are based off of probaility estimates. That is, each observation is given an explicit probability of belonging in each class. In our solution, we can use `predict_proba` to see the predicted probabilities for the first observation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.8709127, 0.1290873]])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get predicted probabilities\n",
    "logit.predict_proba(features_test)[0:1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see the classes using `classes_`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logit.classes_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example, the first observation has an ~87% chance of being in the negative class (0) and a 13% chance of being in the positive class (1). By default, scikit-learn predicts an observation is part of the positive class if the probabaility is greater than 0.5 (called the *threshold*). However, instead of a middle ground, we will often want to explicitly bias our model to use a different threshold for substantive reasons. For example, if a false positive is very costly to our company, we might prefer a model that has a high probability threshold. We fail to predict some positives, but when an observation is predicted to be postivie, we can be very confident that the prediction is correct. This trade-off isrepresnted in the true positive rate (TPR) and the false positive rate (FPR).\n",
    "\n",
    "The true positive rate is the number of observations correctly predicted true divided by all true positive observations:\n",
    "$$TPR = \\frac{True Positives}{True Positives+False Negatives}$$\n",
    "\n",
    "The false positive rate is the number of incorrectly predicted positives divided by all true negative observations:\n",
    "$$FPR = \\frac{False Positives}{False Positives+True Negatives}$$\n",
    "\n",
    "The ROC curve represnets the rspective TPR and FPR for every probability threshold. For example, in our solution a threshold of roughly 0.76 has a TPR of 0.63 and an FPR of 0.68 \n",
    "\n",
    "(note from dustin: idk how to set the threshold and don't know where the indexes 116 and 45 came from in the book)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Threshold: 0.7589444399199625\n",
      "True Positive Rate: 0.6308926780341023\n",
      "False Positive Rate: 0.06879361914257229\n"
     ]
    }
   ],
   "source": [
    "print(\"Threshold: {}\".format(threshold[116]))\n",
    "print(\"True Positive Rate: {}\".format(true_positive_rate[116]))\n",
    "print(\"False Positive Rate: {}\".format(false_positive_rate[116]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However if we increase the threshold to ~89% (i.e. increase how certain the model has to be before it predicts an observation as positive) the TPR drops significantly but so does the FPR:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Threshold: 0.8918937866202341\n",
      "True Positive Rate: 0.41223671013039115\n",
      "False Positive Rate: 0.023928215353938187\n"
     ]
    }
   ],
   "source": [
    "print(\"Threshold: {}\".format(threshold[45]))\n",
    "print(\"True Positive Rate: {}\".format(true_positive_rate[45]))\n",
    "print(\"False Positive Rate: {}\".format(false_positive_rate[45]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is because of our higher requirement for beign predicted to be in the positive class has made the model not identify a number of positive obesrvations (the lower TPR), but also reduce the noise from negative observations being predicted as positive (the lower FPR).\n",
    "\n",
    "In addition to being able to visualize the trade-off between TPR and FPR, the ROC curve can also be used as a general metric for a model. The better a model is, the higher the curve and thus the greater the area under the curve. For this reason, it is common to calculate the area under the ROC curve (AUCROC) to judge the overall quality of a model at al possible thresholds. The closer the AUCROC is to 1, the better the model. In scikit-learn we can calculate the AUCROC using `roc_auc_score`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9060181541633875"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# calculate area under curve\n",
    "roc_auc_score(target_test, target_probabilities)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### See Also\n",
    "ROC Curves in Python and R (https://community.alteryx.com/t5/Data-Science-Blog/ROC-Curves-in-Python-and-R/ba-p/138430)\n",
    "The Area Under a ROC Curve (http://gim.unmc.edu/dxtests/roc3.htm)\n",
    "\n",
    "### 11.6 Evaluating Multiclass Classifier Predictions\n",
    "\n",
    "#### Problem\n",
    "You have a model that predicts three or more classes and want to evaluate its performance.\n",
    "\n",
    "#### Solution\n",
    "Use cross-validation with an evaluation metric capable of handling more than two classes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.83653269, 0.8259826 , 0.81308131])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# load libraries\n",
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.datasets import make_classification\n",
    "\n",
    "# generate features matrix and target vector\n",
    "features, target = make_classification(n_samples=10000,\n",
    "                                       n_features=3,\n",
    "                                       n_informative=3,\n",
    "                                       n_redundant=0,\n",
    "                                       n_classes=3,\n",
    "                                       random_state=1)\n",
    "\n",
    "# create logistic regression\n",
    "logit = LogisticRegression()\n",
    "\n",
    "# cross-validate model using accuracy\n",
    "cross_val_score(logit, features, target, scoring='accuracy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Discussion\n",
    "When we have balanced classes (e.g. a roughly equal number of observations in each class of a target vector), accuracy is--just like in the binary class setting--a simple and interpretable chocie for an evaluation metric. Accuracy is the number of correct predictions divided by the number of observations and works just as well in the multiclass as binary setting. However, when we have imbalanced classes (a common scenario), we should be inclined to use other evaluation metrics.\n",
    "\n",
    "Many of scikit-learn's built-in metrics are for evaluating binary classifiers. However, many of these metrics can be extended for use when we have more than two classes. Precision, recall, and F1 scores are useful metrics that we have already covered in detial in previous recipes. While all of them were originally designed for binary classifiers, we can apply them to multiclass settings by treating our data as a set of binary classes. Doing so enables us to apply the metrics to each class as if it were the only class in the data, and then aggregate the evaluation scores for all the classes by averaging them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.83613125, 0.82562258, 0.81293539])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# cross-validate model using macro averaged F1 score\n",
    "cross_val_score(logit, features, target, scoring='f1_macro')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this code, `_macro` refers to the method uses to average the evaluation scores from the classes:\n",
    "`macro`\n",
    "* Calculate mean of metric scores for each class, weighting each class equally\n",
    "`weighted`\n",
    "* Calculate mean of metric scores for each class, weighting each class proportional to its size in the data\n",
    "`micro`\n",
    "* Calculate mean of metric scores for each observation-class combination."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 11.7 Visualizing a Classifier's Performance\n",
    "#### Problem\n",
    "Given predicted classes and true classes of the test data, you want to visually compare the model's quality\n",
    "\n",
    "#### Solution\n",
    "Use a confusion matrix, which compares predicted classes and true classes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbYAAAEmCAYAAAAOb7UzAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3XeYXVXZ/vHvHSaVFCCVEkCBgIL0LgJBWuhNo1IFzY/ia0FRUKT+EFBEBYyQiIiEVzAgGpIAQqhGEUJNohhqIJJMCC2QQpKZ5/1j7wknw5Qzk1lzxj3357rONWe3tZ6Z2XOeWWvvvZYiAjMzs6LoUukAzMzM2pITm5mZFYoTm5mZFYoTm5mZFYoTm5mZFYoTm5mZFYoTm1krSOop6U5J70oavxrlHCvpL20ZWyVIukvSiZWOwwyc2KzgJH1J0jRJ70uam38A79EGRR8DDAb6R8TnWltIRNwcEfu3QTyrkLS3pJD0x3rrt8nXP1hmORdIGtfcfhExIiJubGW4Zm3Kic0KS9KZwM+BH5EloQ2B0cDhbVD8RsCsiFjRBmWl8gawu6T+JetOBGa1VQXK+HPEOhSfkFZIkvoBFwFnRMQfI2JRRCyPiDsj4qx8n+6Sfi7p9fz1c0nd8217S5oj6duS5uetvS/n2y4EzgNG5i3BU+q3bCRtnLeMqvLlkyS9JOk9SS9LOrZk/V9Ljttd0uN5F+fjknYv2fagpIslTc3L+YukAU38GJYBfwK+kB+/BvB54OZ6P6tfSHpN0kJJT0j6TL7+QOD7Jd/nMyVxXCJpKrAY+Hi+7iv59l9Juq2k/MslTZGksn+BZqvBic2KajegB3BHE/v8ANgV2BbYBtgZOLdk+xCgH7A+cArwS0lrR8T5ZK3AWyOid0Rc31QgktYErgJGREQfYHfg6Qb2WweYlO/bH7gSmFSvxfUl4MvAIKAb8J2m6gZ+B5yQvz8AmAm8Xm+fx8l+BusA/wuMl9QjIu6u931uU3LM8cAooA8wu1553wa2zpP2Z8h+dieGx++zduLEZkXVH1jQTFfhscBFETE/It4ALiT7wK6zPN++PCImA+8Dm7cynlpgK0k9I2JuRMxsYJ+Dgecj4qaIWBERvweeAw4t2eeGiJgVEUuAP5AlpEZFxN+AdSRtTpbgftfAPuMi4s28zp8C3Wn++/xtRMzMj1ler7zFwHFkiXkc8D8RMaeZ8szajBObFdWbwIC6rsBGrMeqrY3Z+bqVZdRLjIuB3i0NJCIWASOBU4G5kiZJ2qKMeOpiWr9keV4r4rkJ+BownAZasHl367/y7s93yFqpTXVxArzW1MaIeAx4CRBZAjZrN05sVlR/B5YCRzSxz+tkN4HU2ZCPdtOVaxHQq2R5SOnGiLgnIvYD1iVrhY0tI566mP7Typjq3AScDkzOW1Mr5V2F3yO79rZ2RKwFvEuWkAAa6z5ssltR0hlkLb/Xge+2PnSzlnNis0KKiHfJbvD4paQjJPWS1FXSCEk/znf7PXCupIH5TRjnkXWdtcbTwJ6SNsxvXDmnboOkwZIOy6+1fUDWpVnTQBmTgWH5IwpVkkYCnwQmtjImACLiZWAvsmuK9fUBVpDdQVkl6Tygb8n2amDjltz5KGkY8P/JuiOPB74rqckuU7O25MRmhRURVwJnkt0Q8gZZ99nXyO4UhOzDdxrwLDAdeDJf15q67gVuzct6glWTUReyGypeB94iSzKnN1DGm8Ah+b5vkrV0DomIBa2JqV7Zf42Ihlqj9wB3kT0CMJuslVvazVj38Pmbkp5srp6863cccHlEPBMRz5PdWXlT3R2nZqnJNyqZmVmRuMVmZmaF4sRmZmaF4sRmZmaF4sRmZmaF0tTDqxXVc7uv+a4W4+3Hr6l0CGbWQfSooqzxRt1iMzOzQnFiMzOzQnFiMzOzQnFiMzOzQnFiMzOzQnFiMzOzQnFiMzOzQnFiMzOzQnFiMzOzQnFiMzOzQnFiMzOzQnFiMzOzQnFiMzOzQnFiMzOzQnFiMzOzQnFiMzOzQnFiMzOzQnFiMzOzQnFiMzOzQnFiMzOzQnFiMzOzQnFiMzOzQnFiMzOzQnFiMzOzQnFiMzOzQnFiMzOzQnFiMzOzQnFiMzOzQnFiMzOzQnFiMzOzQnFiMzOzQnFiMzOzQnFiMzOzQnFiMzOzQnFi6yCuPf9YZk+5lGnjv79y3XmnH8xjt57Do7eczZ2jz2Ddgf0qGKFVwtRHHuawgw/gkAP34/qxYyodjlWIz4OWcWLrIG6681EOP+OXq6z72Y1T2Hnkpez6hcu465EZnDNqRIWis0qoqanhR5dcxOhrf80dEyZx9+SJvPjCC5UOy9qZz4OWc2LrIKY++SJvvbt4lXXvLVq68n2vnt2JiPYOyypoxvRnGTp0IzYYOpSu3bpx4EEH8+ADUyodlrUznwctV1XpAKxpF5xxKMcesjPvvr+EA0ddVelwrB3Nr65myLpDVi4PGjyY6c8+W8GIrBJ8HrRc0habpIGSrpA0WdL9da8m9h8laZqkaSsWzEwZ2n+NC355J5uN+CG33DWNU0fuWelwrB0FH22hS6pAJFZJPg9aLnVX5M3Av4CPARcCrwCPN7ZzRIyJiB0jYseqAVsmDu2/yx/uepwjPrttpcOwdjR48BDmzZ23cnl+dTWDBg2qYERWCT4PWi51YusfEdcDyyPioYg4Gdg1cZ2FscmGA1e+P3ivrZn1SnUFo7H2tuVWn+LVV19hzpzXWL5sGXdPnsRew/epdFjWznwetFzqa2zL869zJR0MvA5skLjO/0o3XnoSn9lhMwas1ZsX7r6Yi6+dzIF7bMlmGw2itjZ4de5bfP2SWyodprWjqqoqzvnBeZw26ivU1tZwxJFHs+mmm1U6LGtnPg9aTinvtJN0CPAIMBS4GugLXBgRE5o7tud2X/MtgMbbj19T6RDMrIPoUUVZFxeTttgiYmL+9l1geMq6zMzMIP1dkT+W1FdSV0lTJC2QdFzKOs3MrHNLffPI/hGxEDgEmAMMA85KXKeZmXViqRNb1/zrQcDvI+KtxPWZmVknl/quyDslPQcsAU6XNBBY2swxZmZmrZa0xRYRZwO7ATtGxHJgEXB4yjrNzKxzS9pik9QVOB7YMx8C5iHg2pR1mplZ55a6K/JXZNfZRufLx+frvpK4XjMz66RSJ7adImKbkuX7JT2TuE4zM+vEUt8VWSNpk7oFSR8HahLXaWZmnVjqFttZwAOSXgIEbAScnLhOMzPrxFIntr8CmwGbkyW25xLXZ2ZmnVzqrsi/R8QHEfFsRDwTER8Af09cp5mZdWJJWmyShgDrAz0lbQcrR2TuC/RKUaeZmRmk64o8ADiJbO61K0vWLwS+n6hOMzOzNIktIm4EbpR0dETcnqIOMzOzhqS+xjZV0vWS7gKQ9ElJpySu08zMOrHUie0G4B5gvXx5FvDNxHWamVknljqxDYiIPwC1ABGxAj+gbWZmCaVObIsk9QcCQNKuwLuJ6zQzs04s9QPaZwITgE0kTQUGAsckrtPMzDqx1C22TYARwO5k19qeJ30yNTOzTix1YvthRCwE1gb2BcaQTVtjZmaWRPLR/fOvBwPXRsSfgW6J6zQzs04sdWL7j6TrgM8DkyV1b4c6zcysE0udZD5Pdm3twIh4B1iHbCobMzOzJJLeyBERi4E/lizPBeamrNPMzDo3dwuamVmhOLGZmVmhOLGZmVmhOLGZmVmhOLGZmVmhOLGZmVmhOLGZmVmhKCIqHUODlq6gYwZm7WrYtyZUOgTrAI4/aItKh2AdwCUjhqmc/dxiMzOzQnFiMzOzQnFiMzOzQnFiMzOzQnFiMzOzQnFiMzOzQnFiMzOzQnFiMzOzQnFiMzOzQnFiMzOzQnFiMzOzQnFiMzOzQnFiMzOzQnFiMzOzQmk2sUnaVVKv/P0XJf1Y0tD0oZmZmbVcOS22McASSVsD3weqgXFJozIzM2ulchLbishmIz0c+EVE/BTokzYsMzOz1qkqY59Fks4CjgP2ltQF6Jo2LDMzs9Ypp8U2EhBwakTMBTYArkwalZmZWSuV02J7G7giImolbQJsDtyUNiwzM7PWKafF9gjQQ9K6wEPAacBvkkZlZmbWSuUkti4RsRg4GrgmIg4FtkkblpmZWeuUldgk7QR8CZjYguPMzMzaXTkJ6kzgQmBSRMyQ9HGy7kkzM7MOp9mbRyLifuD+kuWXgNNTBmVmZtZazSY2SQOAbwNbAj3q1kfE/gnjMjMza5VyuiLHAa8Aw4DLgXnA0wljMjMza7VyEtvAiLgOWBYRU4ATgZ3ThmVmZtY65TygvTz/Ok/SAcDrgEf3NzOzDqmcxPYjSf2A7wC/BPoCZyWNyszMrJXKuStyQv72WeAzacMxMzNbPY0mNkk/A6Kx7RFxZpKIzMzMVkNTLbYZ7RaFmZlZG2kqsY0DekfEm6UrJfUH3k8alZmZWSs1ldh+AUwBxtdbfzCwC3BGqqAMpj7yMJdfdgm1NbUcefTnOOWroyodkrWDn3xpWz671WDefO8D9rv0QQD69erK6C/vyAbr9GTOW0s4/TfTeHfJ8qYLskJZtvh9nrj1ahbOnQ2IHb/4Dfp/bItKh9VhNfUc254RUT+pQTYX295pwjGAmpoafnTJRYy+9tfcMWESd0+eyIsvvFDpsKwdjP/Hq5ww+tFV1p2x32ZMnfUGe118P1NnvcHp+21aoeisUp65YyxDttieA75/Lft99yr6DN6g0iF1aE0lNjW0MiKisW3WNmZMf5ahQzdig6FD6dqtGwcedDAPPjCl0mFZO3jsxbd4Z/GyVdbt96kh3PaP1wC47R+vsf/W61YiNKuQ5UsX88aLM9h412wUwy5VXenWq3eFo+rYmkpsCyTtUH+lpO2Bt5oqVNIaksatbnCd1fzqaoasO2Tl8qDBg6murq5gRFZJA/p0Z/7CDwCYv/ADBvTpVuGIrD0tWjCP7r37Me1/f859P/kG0265ihUfLK10WB1aU4ntLOB2SedKGpG/fgjcTjMPaEdEDTBQUov+AiWNkjRN0rTrx45pyaGFEg08ZSG5kWzWGdXW1vDOnBf5+KcPYt+zfkFVtx48N+W2SofVoTV680hEPCppV+B/gFPz1TOB3SNibhllvwJMlTQBWFRS7pVN1DkGGAOwdEXjz9AV3eDBQ5g3d97K5fnV1QwaNKiCEVklLXjvAwb1zVptg/p2Z8F7y5o/yAqj11oD6NlvAP033hyA9bf5NP92YmtSk4MgR8S8iPhBRByev75fZlKDbEzJiXkdfUpe1owtt/oUr776CnPmvMbyZcu4e/Ik9hq+T6XDsgq5d/o8jtklG571mF2Gcu/0ec0cYUXSo+/a9Fx7AO9VzwFg/qxn6DvYw/U2pZyxIlslIi4EkNQnWww/+1amqqoqzvnBeZw26ivU1tZwxJFHs+mmm1U6LGsHV5+0PbttOoC1e3fjHxftx5WT/83oe5/nVyfvyMhdN+T1t5dw6m+mVTpMa2fbHfX/eGzcT6ldsYI1+w9mxy99s9IhdWjKbnJMULC0FdmjAevkqxYAJ0TEzHKO78xdkfahYd+a0PxOVnjHH+RntgwuGTGsrJsNypmPDQBJ3VsYwxjgzIjYKCI2IpuFe2wLyzAzM2uRZhObpJ0lTQeez5e3kXR1GWWvGREP1C1ExIPAmq0N1MzMrBzltNiuAg4B3gSIiGeA4WUc95KkH0raOH+dC7zc+lDNzMyaV05i6xIRs+utqynjuJOBgcAfgTvy919uWXhmZmYtU85dka9J2hkISWuQPdc2q7mDIuJt4OurGZ+ZmVmLlJPYTiPrjtwQqAbuy9c1SNKdND1B6WEtjNHMzKxszSa2iJgPfKEFZV7R+nDMzMxWT7OJTdJYGmiBRUSDE4RFxEMlx3YDhuWL/44ITyJlZmZJldMVeV/J+x7AkcBrzR0kaW/gRrIxIwUMlXRiRDzc8jDNzMzKU05X5K2ly5JuAu4to+yfAvtHxL/z44YBvwc+MhWOmZlZWyl75JESHwM2KmO/rnVJDSAiZgFdW1GfmZlZ2cq5xvY2H15j60I2yejZZZQ9TdL1ZONFAhwLPNGaIM3MzMrVZGJTNrvlNsB/8lW1Uf6oyacBZ5A9yybgYWB0K+M0MzMrS5OJLSJC0h0R0ZrrYlXAL+omFs0f7m7pQMpmZmYtUs41tsckbd+KsqcAPUuWe7LqHZZmZmZtrtEWm6SqiFgB7AF8VdKLwCKybsWIiOaSXY/SyUUj4n1JvdoiaDMzs8Y01RX5GLA9cEQry14kafuIeBJA0g7AklaWZWZmVpamEpsAIuLFVpb9TWC8pNfz5XWBka0sy8zMrCxNJbaBks5sbGPdTSFNbH9c0hbA5mRJ8jkPqWVmZqk1ldjWAHqTt9zKJWmfiLhf0lH1Nm0miYj4Y0uDNDMzK1dTiW1uRFzUijL3Au4HDm1gW5BNPGpmZpZEs9fYWioizs+/erZsMzNrd009x/bZ1SlY0jck9VXm15KelLT/6pRpZmbWnEYTW0S8tZplnxwRC4H9gUHAl4HLVrNMMzOzJrVmdP9y1XVlHgTcEBHP0MruTTMzs3KlTGxPSPoLWWK7R1IfoDZhfWZmZmXNoN1i+awA5wEDgZciYrGk/mTdkWZmZskkSWz5rAB/Kp0VICLeBN5MUZ+ZmVmdlF2Rj0raKWH5ZmZmH5GkxZYbDpwq6RVWnRVg64R1mplZJ5cysY1IWLaZmVmDknVFRsRsYCiwT/5+ccr6zMzMIGGikXQ+8D3gnHxVV2BcqvrMzMwgbQvqSOAwsutrRMTrQJ+E9ZmZmSW9xrYsv+0/ACStmbAuK6hZPzus0iFYB7DbJfdXOgTrAC4ZMays/VK22P4g6TpgLUlfBe4Dxiasz8zMLGmLrRZ4BFgIDAPOi4h7E9ZnZmaWNLH1AU4B3gJuAZ5NWJeZmRmQ9nb/CyNiS+AMYD3gIUn3parPzMwM2ue5svnAPLJxIge1Q31mZtaJpXyO7TRJDwJTgAHAVz2clpmZpZbyGttGwDcj4umEdZiZma0iWWKLiLNTlW1mZtYYj91oZmaF4sRmZmaF4sRmZmaF4sRmZmaF4sRmZmaF4sRmZmaF4sRmZmaF4sRmZmaF4sRmZmaF4sRmZmaF4sRmZmaF4sRmZmaF4sRmZmaF4sRmZmaF4sRmZmaF4sRmZmaF4sRmZmaF4sRmZmaF4sRmZmaF4sRmZmaF4sRmZmaF4sRmZmaF4sRmZmaF4sRmZmaF4sRmZmaF4sRmZmaF4sTWQU195GEOO/gADjlwP64fO6bS4ViF+DwwgC/usgHjT9uZ207bmS/tskGlw+nwnNg6oJqaGn50yUWMvvbX3DFhEndPnsiLL7xQ6bCsnfk8MIBNBq7JUduvx/FjpzHy2sfZc9gANlynZ6XD6tCc2DqgGdOfZejQjdhg6FC6duvGgQcdzIMPTKl0WNbOfB4YwMcG9mL6nIUsXVFLTQRPzH6H4VsMrHRYHVryxCZpkKQN616p6yuC+dXVDFl3yMrlQYMHU11dXcGIrBJ8HhjAi/MXsf1Ga9GvZxU9qrqwx6b9GdKve6XD6tCSJTZJh0l6HngZeAh4BbirmWNGSZomaVpnvp4QxEfWSapAJFZJPg8M4OUFi/nt1Nn86vjt+OVx2zKr+n1W1H703LAPVSUs+2JgV+C+iNhO0nDgi00dEBFjgDEAS1c08FfdSQwePIR5c+etXJ5fXc2gQYMqGJFVgs8Dq/Onp+byp6fmAvC1fT5O9cIPKhxRx5ayK3J5RLwJdJHUJSIeALZNWF9hbLnVp3j11VeYM+c1li9bxt2TJ7HX8H0qHZa1M58HVmftXl0BGNK3O/t8YiB3z3CXdFNSttjekdQbeBi4WdJ8YEXC+gqjqqqKc35wHqeN+gq1tTUcceTRbLrpZpUOy9qZzwOrc8XnP8VavbqyoqaWyybP4r2l/ihtiiLS9PhJWhNYQtYqPBboB9yct+Ka1Zm7Is1sVbtdcn+lQ7AO4Knz9ynrInPKFtsgYG5ELAVulNQTGAyUldjMzMxaI+U1tvFAbclyTb7OzMwsmZSJrSoiltUt5O+7JazPzMwsaWJ7Q9JhdQuSDgcWJKzPzMws6TW2U8nuhrwGEPAacELC+szMzNIltoh4Edg1v+VfEfFeqrrMzMzqtHlik3RcRIyTdGa99QBExJVtXaeZmVmdFC22NfOvfRKUbWZm1qQ2T2wRcV3+9cK2LtvMzKw5ya6xSRoIfBXYuLSeiDg5VZ1mZmYp74r8M/AIcB/Zw9lmZmbJpUxsvSLiewnLNzMz+4iUD2hPlHRQwvLNzMw+ImVi+wZZclsiaaGk9yQtTFifmZlZ0ge0fbu/mZm1uxQPaG8REc9J2r6h7RHxZFvXaWZmVidFi+1MYBTw0wa2BeC57c3MLJkUD2iPyr8Ob+uyzczMmpPyAe2jGlj9LjA9IuanqtfMzDq3lM+xnQLsBjyQL+8NPAoMk3RRRNyUsG4zM+ukUia2WuATEVENIGkw8CtgF+BhwInNzMzaXMrn2DauS2q5+cCwiHgLWJ6wXjMz68RSttgekTQRGJ8vHw08LGlN4J2E9ZqZWSeWMrGdARwF7AEI+B1we0QE4DsmzcwsiSSJTdIawD0RsS9we4o6zMzMGpLkGltE1ACLJfVLUb6ZmVljUnZFLgWmS7oXWFS3MiK+nrBOMzPr5FImtkn5y8zMrN2kHN3/xlRlm5mZNSbF6P5/iIjPS5pONujxKiJi67au08zMrE6KFts38q83AI8BryWow8zMrEFtfldkRMzN3/YBrgPGAYcASyNidlvXZ2ZmVirZkFoRcWFEbEn2oPZ6wEOS7ktVn5mZGaQdK7LOfGAe8CYwqB3qMzOzTkzZCFcJCpZOA0YCA4HbgFsj4p9JKisoSaMiYkyl47DK8nlg4POgJVImtsuAWyLi6SQVdAKSpkXEjpWOwyrL54GBz4OWSPkc29mpyjYzM2tMe1xjMzMzazdObB2b+9MNfB5YxudBmZJdYzMzM6sEt9jMzKxQnNjMzKxQnNg6CEknSVqv0nFYxyDpIkn7tuK4vSVNTBGTrR5J60m6rRXH/VrSJ5vZ51RJJ7Q+umLxNbYOQtKDwHciYlqlY7H2IUlkf4O1bVjm3mTn0SFl7l8VESvaqn5rOf8O2p5bbAlJWlPSJEnPSJohaaSkHSQ9JOkJSfdIWlfSMcCOwM2SnpbUU9JnJT0labqk30jqnpd5maR/SnpW0hX5ukMl/SPf/z5Jgyv5fXc2ki6XdHrJ8gWSvi3pLEmP57+rC/NtG0v6l6TRwJPAUEm/zc+P6ZK+le/32/y8QNJOkv6Wn0ePSeojqYekG/JjnpI0vIG41pH0p7z+RyVtXRLfGEl/AX7XDj+iTqeJc2JGvnySpPGS7gT+IqmLpNGSZkqaKGlyye//QUk75u/fl3RJfi48Wve3npf/nfz9pvnnwDOSnpS0iaTekqbky9MlHd7uP5T2FBF+JXoBRwNjS5b7AX8DBubLI4Hf5O8fBHbM3/cgm+5nWL78O+CbwDrAv/mwpb1W/nXtknVfAX5a6e+9M72A7YCHSpb/CZxAdnu2yP6BnAjsCWwM1AK75vvuANxbcmzd7/S3wDFAN+AlYKd8fV+ygRW+DdyQr9sCeDU/b/YGJubrrwbOz9/vAzydv78AeALoWemfXVFfjZwTewIz8uWTgDnAOvnyMcDk/FwZArwNHJNvK/1sCODQ/P2PgXNLfqffyd//Azgyf98D6JWfM33zdQOAF+o+M4r4SjbyiAEwHbhC0uVkH2xvA1sB92a9UKwBzG3guM2BlyNiVr58I9ksCdcAS4FfS5qUlwmwAXCrpHXJPghfTvPtWEMi4ilJg/JrpAPJfs9bA/sDT+W79QY2I0tAsyPi0Xz9S8DHJV0NTAL+Uq/4zYG5EfF4XtdCAEl7kCUuIuI5SbOBYfWO3YPsnysi4n5J/SX1y7dNiIglq//dW0MaOSderbfbvRHxVv5+D2B8ZN3S8yQ90EjRy/jw7/4JYL/SjZL6AOtHxB15HEvz9V2BH0nak+wfq/WBwWQD1BeOE1tCETFL0g7AQcClwL3AzIjYrZlD1Uh5KyTtDHwW+ALwNbL/xK8GroyICfk1lgva5juwFriN7L/uIcAtZC2zSyPiutKdJG0MLKpbjoi3JW0DHED2z8vngZNLD6GBmehp5BwpY5+6shY1sM3aVv1zor7S30E5v0+A5ZE3u4AaPvoZ3lg5x5Il2B0iYrmkV8hac4Xka2wJ5f+tLY6IccAVwC7AQEm75du7Stoy3/09sslZAZ4DNpa0ab58PNl8dr2BfhExmaxrctt8ez/gP/n7E1N+T9aoW8j+2TiG7APtHuDk/HeGpPUlfWTaJkkDgC4RcTvwQ2D7ers8B6wnaad8/z6SqoCHyT6skDQM2JCsm7pU6T57AwvqWnzWLuqfE035K3B0fq1tMFmXcovlv985ko4AkNRdUi+yz4j5eVIbDmzUmvL/W7jFltangJ9IqgWWA6cBK4Cr8i6hKuDnwEyyayrXSloC7AZ8GRiff4g9DlxLdo3tz5J6kP1n9q28ngvyff8DPAp8rF2+O1spImbm3UD/iWwW+bmSPgH8Pe92fh84juy/7FLrAzdIqvsn85x65S6TNBK4WlJPYAmwLzCa7HyZTnZOnRQRH+R11bkgL/tZYDH+p6dd1T8n8tZ6Y24n64mZAcwiu072biurPh64TtJFZJ87nwNuBu6UNA14muwfpsLy7f5mZh2ApN4R8b6k/sBjwKcjopDXwFJzi83MrGOYKGktshvALnZSaz232MzMrFB884iZmRWKE5uZmRWKE5uZmRWKE5tZCUk1ysbrnJGP5ddrNcpaOdK+pMMknd3EvmuVji3YgjpWjhHYwLYT8u9jprLxRevGElw5DqVZETmxma1qSURsGxFbkQ1fdGrpRmVa/HcTERMi4rImdlkLaHFia4ykEWQP8e8fEVuSPfjd2ueizP6rOLGZNe4RYFM1PCL//pL+no+WPr5khJEDJT0n6a/AUXUFKRvN/Zr8/WBJd+Sjrz8jaXfgMmCTvLX4k3y/j8wOkK//gaR/S7qPbCzJhpxDNiju65CNGRgRY+vvJOm8vI5CMHeBAAACjElEQVQZykb8V77+6/pwFolb8nV75fE9rWxGgT71yzPrCJzYzBqQj/gygmwga8gSyO8iYjuyMf7OBfaNiO2BacCZ+YgwY4FDgc+QjRHYkKvIRn7fhqwlNRM4G3gxby2eJWl/skGTdyYbOm0HSXvmY49+gWz0+KOAnRqpYyuyQXKbc01E7JS3UHsCdfO4nQ1sFxFb82Gr9TvAGRGxbf79eRBl65Cc2MxW1VPS02TJ6lXg+nx96Yj8uwKfBKbm+55INvbeFmSzMjyfD1Q7rpE69gF+BRARNRHRUBfh/nw4O8CTedmbkSWUOyJicT4u4ITV+m5huLK5/KbncdWNXfos2fyAx5EN2QUwFbhS0tfJptfx5JjWIXnkEbNVLclbJCvlvXP1R2K/NyK+WG+/bWl4JP7WEA3PDvDNMuuYSTbX2/2NVpC1MEeTzfX1mqQL+HDE94PJ5g87DPihpC0j4jJl0yUdBDwqad+IKPSYg/bfyS02s5Z7FPh03ewLknrlI+w/B3xM0ib5fl9s5PgpZANiI2kNSX1ZdXYHaHx2gIeBI5XNst6HrNuzIZcCP5Y0JD++e97SKlWXxBbk9dTN2NwFGBoRDwDfJbuxpbekTSJiekRcTtai3aKpH5JZpbjFZtZCEfGGpJOA30vqnq8+N59/bxQwSdICsqlItmqgiG8AYySdQjba/2kR8XdJUyXNAO7Kr7N9ZHaAiHhS0q1kI7TPJrvBpaEYJyub/uS+/IaQAH5Tb593JI0lu474CtksEpBNgDtO2QwUAn6W73uxsilPashmhL6rZT85s/bhsSLNzKxQ3BVpZmaF4sRmZmaF4sRmZmaF4sRmZmaF4sRmZmaF4sRmZmaF4sRmZmaF8n8X7jZZtfFghQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# load libraries\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from sklearn import datasets\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import confusion_matrix\n",
    "import pandas as pd\n",
    "\n",
    "# load data\n",
    "iris = datasets.load_iris()\n",
    "\n",
    "# create feature matrix\n",
    "features = iris.data\n",
    "\n",
    "# create target vector\n",
    "target = iris.target\n",
    "\n",
    "# create list of target class names\n",
    "class_names = iris.target_names\n",
    "\n",
    "# split into training and test sets\n",
    "features_train, features_test, target_train, target_test = train_test_split(features, target, random_state=1)\n",
    "\n",
    "# create logistic regression\n",
    "classifier = LogisticRegression()\n",
    "\n",
    "# train model and make predictions\n",
    "target_predicted = classifier.fit(features_train, target_train).predict(features_test)\n",
    "\n",
    "# create confusion matrix\n",
    "matrix = confusion_matrix(target_test, target_predicted)\n",
    "\n",
    "# create pandas dataframe\n",
    "dataframe = pd.DataFrame(matrix, index=class_names, columns=class_names)\n",
    "\n",
    "# create heatmap\n",
    "sns.heatmap(dataframe, annot=True, cbar=None, cmap=\"Blues\")\n",
    "plt.title(\"Confusion Matrix\"), plt.tight_layout()\n",
    "plt.ylabel(\"True Class\"), plt.xlabel(\"Predicted Class\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Discussion\n",
    "#### See Also\n",
    "* Confusion Matrix (https://en.wikipedia.org/wiki/Confusion_matrix)\n",
    "* scikit-learn documentation: Confusion Matrix (http://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html)\n",
    "\n",
    "### 11.8 Evaluating Regression Models\n",
    "#### Problem\n",
    "You want to evaluate the performance of a regression model.\n",
    "\n",
    "#### Solution\n",
    "Use mean squared error (MSE):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-1718.22817783, -3103.4124284 , -1377.17858823])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# load libraries\n",
    "from sklearn.datasets import make_regression\n",
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.linear_model import LinearRegression\n",
    "\n",
    "# generate features matrix, target vector\n",
    "features, target = make_regression(n_samples = 100,\n",
    "                                   n_features = 3,\n",
    "                                   n_informative = 3,\n",
    "                                   n_targets = 1,\n",
    "                                   noise = 50,\n",
    "                                   coef = False,\n",
    "                                   random_state = 1)\n",
    "\n",
    "# create a linear regression object\n",
    "ols = LinearRegression()\n",
    "\n",
    "# cross-validate the lienar regression using (negative) MSE\n",
    "cross_val_score(ols, features, target, scoring='neg_mean_squared_error')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another common regression metric is the coefficient of determination, $R^2$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.87804558, 0.76395862, 0.89154377])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cross_val_score(ols, features, target, scoring='r2')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Discussion\n",
    "MSE is one of the most common evaluation metrics for regression models. Formally, MSE is:\n",
    "\n",
    "$$\n",
    "MSE = \\frac{1}{n} * \\sum_{i=0}^n{(\\hat y_i - y_i)^2}\n",
    "$$\n",
    "\n",
    "where $n$ is the number of observations\n",
    "$y_i$ is the true value of the target we are trying to predict for observation $i$\n",
    "$\\hat y_i$ is the model's predicted value\n",
    "\n",
    "MSE is a measurement of the squared sum of all distances between predicted and true values.\n",
    "\n",
    "The higher the value of MSE, the greater the total squared error and thus the worse the model. Ther are a number of mathematical benefits to squaring the error term, including that it forced all error alues to be positive, but one often unrealized implication is that squaring penalizes a few large errors mroe than many small errors, even if the absolute value of the errors is the same. \n",
    "\n",
    "For example, imagine wo models, A and B, each with two observations:\n",
    "* Model A has errors of 0 and 10 and thus its MSE is $0^2 + 10^2 = 100$.\n",
    "* Model B has two errors of 5 each, and thus its MSE is $5^2 + 5^2 = 50$\n",
    "\n",
    "Both models have the same total error, 10; however, MSE would consider Model A (MSE = 100) worse than Model B (MSE=50). In practice this implicatino is rarely an issue (and indeed can be theoretically beneficial) and MSE works perfectly fine as an evaluation metric\n",
    "\n",
    "One important note: by default in scikit-learn arguments of the `scoring` parameter assume that higher values are better than lower values. However, this is not the case for MSE, where higher values mean a worse model. For this reason, scikit-learn looks at the negative MSE using the `neg_mean_squared_error` argument\n",
    "\n",
    "A common alternative regression evaluation metric is $R^2$, which measures the amount of variance in the target vector that is explained by the model:\n",
    "$$\n",
    "R^2 = 1 - \\frac{\\sum_{i=1}^n{(y_i - \\hat y_i)^2}}{\\sum_{i=1}^n{(y_i - \\bar y)^2}}\n",
    "$$\n",
    "\n",
    "where $y_i$ is the true target value of the ith observation\n",
    "\n",
    "$\\hat y_i$ is the predicted value for the ith observation\n",
    "\n",
    "and $\\bar y$ is the mean value of the target vector. \n",
    "\n",
    "The closer to 1.0, the better the model.\n",
    "\n",
    "#### See Also\n",
    "* Mean squared error (https://en.wikipedia.org/wiki/Mean_squared_error)\n",
    "* Coefficient of determination (https://en.wikipedia.org/wiki/Coefficient_of_determination)\n",
    "\n",
    "### 11.9 Evaluating Clustering Models\n",
    "#### Problem\n",
    "You have used an unsupervised learning algorithm to cluster your data. Now you want to know how well it did.\n",
    "\n",
    "#### Solution\n",
    "The short answer is that you probably can't, at least not the way you want.\n",
    "\n",
    "That said, one option is to evaluate clustering using silhouette coefficients, which measure the quality of the clusters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8916265564072142"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "from sklearn.metrics import silhouette_score\n",
    "from sklearn import datasets\n",
    "from sklearn.cluster import KMeans\n",
    "from sklearn.datasets import make_blobs\n",
    "\n",
    "# generate feature matrix\n",
    "features, _ = make_blobs(n_samples = 1000,\n",
    "                         n_features = 10,\n",
    "                         centers = 2,\n",
    "                         cluster_std = 0.5,\n",
    "                         shuffle = True,\n",
    "                         random_state = 1)\n",
    "\n",
    "# cluster data using k-means to predict classes\n",
    "model = KMeans(n_clusters=2, random_state=1).fit(features)\n",
    "\n",
    "# get predicted classes\n",
    "target_predicted = model.labels_\n",
    "\n",
    "# evaluate model\n",
    "silhouette_score(features, target_predicted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Discussion\n",
    "Supervised model evaluation compares predictsion (e.g. classes or quantitative values) wit hteh corresponding true values in the target vector. However, the most common motivation for using clustering methods is that you data doesn't have a target vector. There are a number of clustering evaluation metrics that require a target vector, but again, using unsupervised learning approaches like clustering when you ahve a target vector available to you is probably handicapping yourself unnecessarily.\n",
    "\n",
    "While we cannot evaluate predictions versus true values if we don't have a target vector, we can evaluate the nature of the clusters themselves. Intuitively, we can imagine \"good\" clusters having very small distances between the different clusters (i.e., well separated clusters). Silhouette coefficients provide a single value measuring both traits. Formally, the ith observation's silhouette coefficient is:\n",
    "\n",
    "$$\n",
    "s_i = \\frac{b_i - a}{max(a_i, b_i)}\n",
    "$$\n",
    "\n",
    "where $s_i$ is the silhouette coefficient for observation i\n",
    "$a_i$ is the mean distance between i and all observations of the same class\n",
    "$b_i$ is the mean distance between i and all observations of a different class\n",
    "\n",
    "Silhouette coefficients range between -1 and 1, with 1 indicating dense, well-separated clusters.\n",
    "\n",
    "#### See Also\n",
    "* scikit-learn documentation: silhouette_score (http://scikit-learn.org/stable/modules/generated/sklearn.metrics.silhouette_score.html#sklearn.metrics.silhouette_score)\n",
    "\n",
    "### 11.10 Creating a Custom Evaluation Metric\n",
    "#### Problem\n",
    "#### Solution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Discussion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### See Also\n",
    "* . scikit-learn documentation: make_scorer (http://scikit-learn.org/stable/modules/generated/sklearn.metrics.make_scorer.html#sklearn.metrics.make_scorer)\n",
    "\n",
    "### 11.11 Visualizing the Effect of Training Set Size\n",
    "#### Problem\n",
    "You want toe valuate the effect of the number of observations in your training set on some metric (accuracy, F1, etc)\n",
    "\n",
    "#### Solution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzs3XeclNXZ+P/PNTM7O9t7oYMKKiBEXLtRY0FUlKjxMaRYMBZCyeMvajTfPLZoNCbGggZLYleIiTGSCChoCFiiYI0gZUWEZQvb++y08/tj9p7MLltmd2e2sNf79ZrXTrnn3GdmZ+5rzrnPuY4YY1BKKaUGG9tAV0AppZTqiAYopZRSg5IGKKWUUoOSBiillFKDkgYopZRSg5IGKKWUUoOSBiil+oGIrBKRywa6HkoNJRqg1AFNRHaJyBkDXQ9jzNnGmGdiUbaIpIrIAyKyW0QaRKSw9XZ2LPanVH/RAKVUH4mIYwD37QTeBKYAs4BU4ASgEjimF+UN2GtRqj0NUGrYEpHZIvKJiNSIyLsiMi3ssZtE5EsRqReRLSJyQdhjl4vIOyJyv4hUAbe13ve2iPxWRKpF5CsROTvsOetE5Edhz+9q2wkisr5132tF5BEReb6Tl3EpMBa4wBizxRgTMMbsM8b80hizsrU8IyKHhJX/tIjc2Xr9VBEpEpGfiUgp8JSIfCEis8O2d4hIhYjMaL19XOv7VSMin4rIqX35PyjVGQ1QalhqPdg+CVwDZAGPAStEJL51ky+BbwJpwO3A8yIyIqyIY4GdQC5wV9h924Bs4F7gjyIinVShq21fBD5orddtwA+7eClnAKuNMQ3dv+pO5QOZwDjgamAZMDfs8bOACmPMRyIyCngNuLP1OdcDL4tITh/2r1SHNECp4eoq4DFjzPvGGH/r+aEW4DgAY8yfjTHFrS2SPwE7aNtlVmyMWWKM8Rljmlvv+9oY84Qxxg88A4wA8jrZf4fbishY4GjgFmOMxxjzNrCii9eRBZT06h34rwBwqzGmpfW1vAicLyKJrY9/r/U+gB8AK40xK1vfmzXAJuCcPtZBqf1ogFLD1Tjgp63dVDUiUgOMAUYCiMilYd1/NcBUgq0dy54Oyiy1rhhjmlqvJney/862HQlUhd3X2b4slQSDW1+UG2PcYfUpBL4AzmsNUufz3wA1Dri43ft2UhTqoNR+9ISoGq72AHcZY+5q/4CIjAOeAE4H3jPG+EXkEyC8uy5WywCUAJkikhgWpMZ0sf1a4E4RSTLGNHayTROQGHY7HygKu93Ra7G6+WzAltagBcH37TljzFXdvA6l+kxbUGo4iBMRV9jFQTAAXSsix0pQkoicKyIpQBLBg3Y5gIhcQbAFFXPGmK8JdpndJiJOETkeOK+LpzxHMGi8LCKHiYhNRLJE5OciYnW7fQJ8T0TsIjILOCWCqiwHZgLz+W/rCeB5gi2rs1rLc7UOtBjdw5eqVLc0QKnhYCXQHHa5zRizieB5qIeBaqAQuBzAGLMFuA94DygDjgDe6cf6fh84nmD33Z3AnwieH9uPMaaF4ECJrcAaoI7gAIts4P3WzX5CMMjVtJb9t+4qYIwpIfj6T2jdv3X/HmAO8HOCAXwPcAN6LFExILpgoVKDm4j8CdhqjLl1oOuiVH/SXz1KDTIicrSIHNzaXTeLYIul21aPUgcaHSSh1OCTD/yV4BDyImC+Mebjga2SUv1Pu/iUUkoNStrFp5RSalAacl182dnZZvz48QNdDaWUUr304YcfVhhjuk2PNeQC1Pjx49m0adNAV0MppVQvicjXkWynXXxKKaUGJQ1QSimlBiUNUEoppQYlDVBKKaUGJQ1QSimlBiUNUEoppQYlDVBKKaUGJQ1QSimlBqVhF6CMMVRVVeH1ege6KkoppboQswAlIk+KyD4R+byTx0VEHhKRQhH5TERmxKou4fx+PzU1NRQVFVFeXo7f7++P3SqllOqhWLagngZmdfH42cDE1svVwNIY1qUNEcEYQ319Pbt376a6uppAINBfu1dKKRWBmOXiM8asF5HxXWwyB3jWBNf7+LeIpIvIiNalpvuNMYaamhpqa2ux2+0sWrQIEWmzzQ9+8AMuuugiysrKuPbaa/cr40c/+hHnnnsuu3bt4rrrrmvzmMPhYP78+Zx88slUVFSwYcMGcnJyyMrKwm63U1NTw7hx40hNTaWwsJCVK1dSXV2N3+8nJSWF5ORkLr74YkaNGkVxcTHbt28HoKmpiebmZpqamjj//PNJS0vj3XffZcOGDTidzjaXiy++mJSUFD799FM+++wzPB4Pfr8fr9eLz+fjmmuuwel08s477/DJJ59gs9mw2WyICCLClVdeCcCKFSt4//33cbvdNDc34/f7iY+P5+GHHwbg8ccf54MPPgi9dpvNRkZGBvfee2/o8c8/DzaoA4EAfr+f3Nxcbr/9dgDuueceduzYgcPhwG63Y7fbGT9+PNdffz0ADz74IEVFRfj9fgKBAIFAgEMOOYTFixcDcNttt7Fv37427/8RRxzB/PnzAbjxxhupq6sLPSYiHH300cybNw+AO+64A7/fj91uR0Tw+XzMmDGDOXPmEAgEWLx4MT6fD2MMcXFxOBwOTjvtNL797W/T0tLCr371K4wx+P1+fD4ffr+fmTNncuaZZ1JdXc0vfvELjDGhuhtjuPjii5k5cyZlZWXccccdOBwO4uLiQuXPmTOHgoICiouLefrpp0P/E8ucOXOYPHkyO3bs4Mknn8Tn8+Hz+QgEAjgcDubNm8fkyZPZunUry5cvx263A+D1evF4PFxzzTVMmDCBd999lz/84Q+ISJs6/PSnP2Xs2LFs2rSJFStWhF6Xdbn55pvJzs5mzZo1vPrqq/t9N+6++25SU1N57bXXWLNmTehzZf298847cTqdrFixgrfffjv0POs9uu+++wB48sknWbduXahr3ul0kpqaypIlSwBYtmwZW7ZsQURC721KSgo333xz6LP3xRdfYIwJXfLz8/l//+//AfDAAw+wc+fONnUbN24cP/nJTwC47777KC4ubvP8iRMn8uMf/xiAX/7yl1RWVrZ57dOmTWPevHmICL/+9a9xu904HI7Q/3Dq1Kmcd955ANx77714vd5Q2YFAgIKCAs455xz8fj+33norNpsNu90e+mwcd9xxnHLKKTQ1NfHEE0/g8XjweDy0tLTg8XiYOXMmp512GtXV1dx+++2hz7X13TrvvPM47rjjKC0t5YEHHmizb4CLL76YY489lq+//ppHHnkk9LmbN28ehx122H7/61gYyGSxo4A9YbeLWu/bL0CJyNUEW1mMHTs2qpX46quvKCws5Mwzz6S2tpba2tr9tikuLmbnzp1UVFRQU1PT6eNFRUX7Pe71eikqKmL37t188MEHoQ90uIceeogTTzyRjz/+mPvuu4+kpCTsdjuNjY34/X4OP/xwjDH87W9/484779zv+WPHjuXggw/mvffe4/7779/v8WnTppGTk8NLL73E448/vt/jZ5xxBsnJyfz1r3/l+eef3+/xM888E7vdzj//+U9WrlxJfHw88fHxOBwOEhIS2LMn+G/cvn07mzdvBgh90NPT09m9ezcA77//fiiAWV+WUaNGhR7fsWMHmzdvbnMQ3LNnDxdffDEAb7zxBoWFhaEvmYhQUVHBnDlzAPjggw8oLi7e7/3/+utgXsoPP/yQ6urq0GPWQdx6fPny5TQ0NISCkMPhoLKykm984xsYY1i/fn1ov1b9UlJSmD59Og0NDTzxxBNtDgA2m43U1FQmTpxIZWUl69evbxP4RYQjjjiCSZMmsWfPHtavXx8KMD6fD6/XS05ODtnZ2WzevDl0ELHeX4CMjAwSEhL47LPPWLZsWWjf1oG6oKCAxMRENm7cyBNPPBHq0nY6ncTFxXHcccchIhQWFvKf//wn9J5ZdZg9ezY+n49//vOf/OEPfwi9Lms/F154IQ0NDXz66af861//2u+zU1hYSEZGBhs3bmT16tWh4GH9veKKK3C5XKxfv56//vWvoeeJCHFxcSxcuBARYfv27Wzfvh2HI3jI8vl8xMfHs2vXLgBef/113nzzTYDQe5ybm8t3v/tdANatW8emTZvavPdjx45l7ty5AKxfv57Nmze3CUCHHnpoKICsXr06FMCsA/WMGTM4++yzAXjrrbcoKfnvocv64futb30LgGeeeYaqqqrQ5wbgnHPOYcqUKQDcf//9eDyeUPk2m40LL7yQww8/HJ/PxzPPPIMxJvTdCAQCXH755YwZM4a6urrQjzybzRb6cZGYmMj48eMpLS3ltddeC/14soJ/eno6eXl5fPXVVyxfvrzNeyMiHHzwweTm5rJ582b+8pe/hF7bUUcdxYQJE4iPj9/v/x1tMV2wsLUF9Q9jzNQOHnsNuNsY83br7TeBG40xH3ZVZkFBgelLNnOfz8eePXsoKyvj8ccfZ8WKFWRlZbFixQqcTiejRo0KfQmixePx0NzcTFVVFV9++SUVFRVUVFTg8/lIT0/niCOOIDs7O/TrMDExkbi4OIwxuN3uUDlVVVXs3r0bEcHlcuFyuUhMTCQrKwun04nNFuyxtX4dW3/T0tKw2WzU19fT3Nwc+gVm/VJOSEgI1dPj8YRaN9ZBJD09PXRAsp5nHaQs7VudFuvLbl23/oZ/0cO/FOHPCb+EPy98f+3/RiJ82/BfjeH7CwQCbYJJ+8DS0WuzLu1fW2f1bF/n9vUIv97Ze2v9z9vXLbyOHdU1/DWF/+3ufWn/P+votXT1v+jos9CZ8NfU/v1v3xINv97Z+9HZZ8Uqr7M6tt82fLtIX2tHn+eudPc5sb6f1o8lt9tNfHx86DgQfgl/Xkffx57uH4LHqPCye0pEPjTGFHS33UC2oIqAMWG3RwPFnWwbNbW1tSxZsoRly5bh9/u55JJLuPLKK0NdBk6nM+r7tIJJRkYGBx10UKiLzAoW1gfL6XS2Oei3N378eI488sg2B0+llDpQDWSAWgEsFJHlwLFAbX+cf9q1axfPPvssZ511Ftdeey2jRo0C/nu+JNZEhISEhFCrpTfP7yqIKaXUgSJmAUpElgGnAtkiUgTcCsQBGGMeBVYC5wCFQBNwRazqEm769OmsWLGC/Pz88LqSk5PTpyarUkqp6IrlKL653TxugAWx2n9XRowY0ab/NSEhgcTExIGoilJKqU4M+yaDiJCdnT3Q1VBKKdXOsA5QIkJmZmbUR+0ppZTqu2EdoBwOB6mpqQNdDaWUUh0YlgHKmr+Qm5urQ7WVUmqQGpYBCiA5OblfZkIrpZTqnWEXoGw2G06nk6ysrIGuilJKxVSkmSsi5fF4qKqqwufzRa3Mrgy70QE2m43Ro0cPdDWUUkOAlTh1qM2RNMbQ2NgYyv+XkJBASkoKCQkJPX4tPp+PhoYG6urq8Pv9GGNwuVz9Mrhs2AUopZSKhNfrpaSkhEAgwMiRI2OSBi3ajDE0NDRQVVXVJjehtfoBBFOvJScnk5SUtF+uPutvIBCgubmZuro6Wlpa2uQr7M/z9hqglFKqnebmZkpLS0MH5b179zJixAhcLlefyrUykocHj/ZJfLvLydlZufX19aG17Trq1rPua25uxu12U15evl9SYUtHyYYHggYopZQKU1tbS1VV1X4ZzktKSsjNzSUpKSnisowxeL1e3G53m1aM1SJp/9d6jt1ux+VykZCQgMvlIi4urk3Q8Hq9oYvH46GpqalH55u6CzwDFZDa0wCllFIED8oVFRU0NDR02gLZt28fmZmZpKWldVqO3++nsbGxTUCynt/+emeBIrwMS1xcXKj11VnL50CjAUopNez5/X5KS0vxeDxdHvSNMaFRbJmZmaFAYQWU+vr6/c7Z9EV4GR6Pp8P7D2QaoJRSg4IxhpaWFlpaWnC73aFVda1FMq0FM/syoi58sb/wpetrampCK91GUoY1os3lclFXVxdaDXegz9n0h8rKSvLy8vplXxqglFJ9Yp376EngsAYLWMGoubkZr9fbYddV+Cq6Xa1i2/45nW3T0WM9DSjWMO7GxsZBHZSKi4txOp1kZmb2eah8S0sL9913H2+99RZr167loIMOilItO6cBSqlhzhgTGlLs8XjIzMwkKSkpouHELS0tlJWV4fP5EJFQa8fpdBIXF0dcXFwoGHk8HlpaWvD5fPj9/g6DUWfnfrrbpqPn9OaxnhiMAQmgqqqKVatW8Y9//IMdO3YAYLfbyc7OJi8vj5ycHHJzcxk7diznnntuREsN7dmzh5/97Gds376dyy67rN8SHchgfZM7U1BQYDZt2jTQ1VAHAI/H02Z01HDj9Xqpq6ujvr6+TRAQEeLj48nJySEuLq7D51rnYurq6jo9UFvva3jrR8WGx+Ph7bff5u9//zvvvvsufr+fyZMnM2vWLBwOB+Xl5ZSVlVFeXs6+ffvYt28fzc3NZGZmcuWVV3LhhRd2+r9eu3Ytv/zlL7Hb7dx+++2cfPLJ5OXl9WkNPRH50BhT0O12Q+1DowFK9ZUxhurqampqanA6neTn5w+bJVcCgQCNjY2h1lJX338RIT09nfT09DZB3O12s2/fvlBWATUwWlpa+Oijj1i/fj1vvPEGtbW1ZGdnc8455zB79uwuu+CMMWzevJklS5bw4YcfMmrUKK699lrOOuusUFegx+Ph/vvv589//jNHHHEEd999N/n5+YiIBqjOaIBSfREIBEK/HsNbDH39wg1m4V141rDnSL/3IoLdbicnJ4f4+HiqqqpCLS4VPY2NjezZswefz0dWVhaZmZkdJrMuLS3lnXfe4e2332bjxo243W7i4+M55ZRTmD17Nsccc0yPfmwZY3jvvfd4+OGH2b59OxMnTmTBggWMHz+em2++mS+++ILvf//7LFy4MNTC0gDVBQ1Qqrd8Ph8lJSX4fL4OZ86npKSQlZUVky6/ng4kCM+lZowhKSkplIE/kvpZI+Lq6+tpaGgI3ddb7QcqqJ4LBAIUFxdTWFjI7t272bNnD19//TW7d++moqJiv+2Tk5PJzMwkOzubjIwMvv76awoLCwEYNWoUJ554IieeeCJHHXVUnzNcBAIB3njjDZYuXcrevXux2+0kJiZy6623cuqpp7bZVgNUFzRAqfYaGhpoaWkhMTERl8vV4QHc7XZTWloaSv7ZEeskf35+fqf98ZEwxoQGBFij1LxeLwAJCQmkpaWRkJDQ6WiypqYmKisr9+tCs4JEcnJym+Vi/H5/KKtAS0sLHo8Hr9erwaQPampqePHFF1m5ciVjx46loKCAo48+msMPPzyiForb7aawsJDt27ezY8cOtm3bRmFhYZuJtxkZGYwdO5axY8cybtw4xowZg9PppLKykqqqqjZ/KysrycnJ4aSTTuKkk05i3LhxMfkh5fV6eeWVV/j4449ZuHAho0aN2m+bAyZAicgs4EHADvzBGHNPu8fHAU8COUAV8ANjTFFXZWqAGvqiOTihpaWF4uLiNsOHnU4nSUlJJCYmEhcXR11d3X6pa7oiIuTk5JCcnBxxPaxcaLW1tZ0Olw4vX0RITU0lNTUVh8PRZWDqrI5W+eHv41AMSLW1tezcubPNxWazcfXVVzN9+vR+rUtFRQXPPfccL7/8Mi0tLRx//PGUl5eHRsMlJSVx5JFHUlBQwKRJk6iurg4NOigrKwtdLy8vD/0vkpKSmDhxIpMmTWLSpEkccsghjB07dsiu5n1ABCgRsQPbgTOBImAjMNcYsyVsmz8D/zDGPCMipwFXGGN+2FW5GqCGtubmZkpKSkKjxPqSIToQCLBnz54OJ1i2D349/Zxbran09HSSk5O7nFfT2NhIZWVlp0k6u9tPfHw8fr+/w67HA9GePXtYs2YNGzduZOfOnVRWVoYeS0xMZMKECaGD/MyZM1m0aBEjRoyIaZ2Ki4t59tlnWbFiBX6/n7POOosrrriCCRMmAFBdXc2HH37Ipk2b2LhxI19//XWb5ycmJpKXl0dubi65ubmMGDEiFJRGjhx5QI0UPVAC1PHAbcaYs1pv3wxgjLk7bJvNwFnGmCIJ/gdrjTFd/qzQADV0tQ8oVisiIyOjV5MIy8rKQkkyY8U6sKSmppKWlhbq3rEGHlRWVg6bwNIXRUVFrF27lrVr17J161YADjvsMCZOnMjBBx/MhAkTOPjgg8nLy0NEaG5u5tlnn+XZZ58F4Pvf/z6XX355nweyGGOora2luLg4dNm2bRtr165FRDjvvPO47LLLul0zbt++fXz99ddkZWWRm5vbo9b2UCci5Ofnk5CQ0JcyBjxAfQeYZYz5UevtHwLHGmMWhm3zIvC+MeZBEbkQeBnINsZUtivrauBqgLFjxx7V/teLGhoqKyv3mzcjIthsNnJycnp08Kmrq6OysrJfA4OI4HK5SElJoba2ttth2kOJ2+3m3//+NyLCoYceGgoUvdXS0sL27dv56KOPePPNN9myJdhxMnXqVM444wzOOOMM8vPzuy2ntLSURx55hFWrVpGdnc2CBQs499xzCQQC1NTUUFVVFTpPU11dHTof6Xa7cbvdoestLS1UVFRQUlLS5jwQQFpaGmeffTY/+MEPIqrTUNK+K9hms7VJHdXTEZnWd8Aabt6Heg14gLqYYOsoPEAdY4xZFLbNSOBhYAKwHrgImGKMqe2sXG1BDU0ej4e9e/d2OanT5XKRk5PT7Uno7sqKtWglAh1ogUCAjz76iFWrVrF27VoaGxtDj6Wnp3PooYe2uWRlZeFwOIiLi8Nut4cOUIFAgN27d/P555+zefNmNm/ezPbt20PLgk+ePDkUlEaOHNmrun7++efcd999/Oc//yEpKanLlnN8fDwulwuXy0V8fHzokpWVxciRIxkxYgQjR44MXQ6k1k/45GiXy0VSUhLx8fGd5jDsaGmRrjidTkaOHNnntEmDIUB128XXbvtkYKsxpsu2tQaooccYw969e9tkY+6MNdw7IyOjw0XbAoEARUVFoYPfcBQIBKiqqsLr9fbql+yXX37JqlWrWLVqFWVlZSQmJnLaaacxa9YsEhMT2bp1K9u2bQuNPOvsvbaSt0KwxQTBAQGTJ09mypQpTJkyhalTp5KTk9O3F9zKGMMbb7zBxx9/TEZGBpmZmWRmZobmDfUkRdOBxHq9LpeLxMREEhISejQIqaqqitra2m6DlN1uZ/To0T1eTLGTOg94gHIQHCRxOrCX4CCJ7xljNodtkw1UGWMCInIX4DfG3NJVuRqghp6e/kqDzs9PlZeXU19f3+c6bd++nT179vCtb32rz78GY6W+vp533nmHvXv3UlJSErpYy0JAcK5MeCvnsMMOY9y4cTgcDurq6ti5cydfffVVm7/79u3Dbrdz3HHHcfbZZ3Pqqad2Oo/G6/Wyc+dOtm/fTm1tLT6fL5RLz7pujOGggw5i6tSpjBs3LioHMNU9ayBPRkZGnwKzMYby8vI2iW872tfo0aP7NP2iXXmDYpj5OcADBIeZP2mMuUtE7gA2GWNWtJ6nuhswBLv4FhhjWroqUwPU0OLz+dizZ0+vusSsL1xGRgapqak0Nzezb9++PnWv+f1+nnvuOZYuXYrf72f69On87Gc/Y9KkSb0uM5qMMXz66af87W9/Y82aNaGWSWZmJiNGjGhzsdvtbN++nW3btrFjx47QtvHx8SQnJ7cZHedyuZgwYQITJkxgypQpnHnmmWRmZg7Ia1R9IyIkJCSQnp7e5wm6FmNMaNBRR/uLxnL37coc+AAVCxqghpbS0tIOP/Q9Ea0sBqWlpdxyyy189NFHnH766Rx77LEsXbqU2tpa/ud//odrr722T+cjSkpKCAQCjBgxosetstraWlauXMkrr7zCzp07SUpKYtasWZx33nkccsgh3R4cfD4fX3/9dahrrr6+PhSQDjroIPLz8wdtS1H9V/hn3RKekqv9aNJoMsZQXFwc+qFj7bOncwIjoQFKDbimpibKysoGxYCC119/nbvvvptAIMCNN97Iueeei4hQV1fH0qVLefnll8nIyGDx4sWcc845Peouqa+v5/e//z0vv/wygUCA+Pj4NsHBug7BQFRXV0dNTU3o+u7du/nXv/6Fx+NhypQpXHDBBcycOfOAzQ2o/sv6nNlsNhITE0lMTAwNQLHZbG3+9se5tUAgwN69e0OTzTMyMkhPT4/6fjRAqQHV1STa/tTQ0MCvf/1rVq1axRFHHMEvf/nLDue4bN26lXvuuYfPP/+cI488kp/85CdMmTKly4NCIBDgtdde46GHHqK2tpaLLrqISZMmhc73fPXVV5SVlXVZP5vNRkZGBqeddhoXXHDBoOlq7A/tM2CED4EG2iT0HYy6yhbS3XMg2BUbnvFksPD7/RQVFZGYmEh2dnZMAqMGKDWgKioq+j3rtZWp3ErAuXv3btatW8e+ffu48sormTdvXpddI4FAgBUrVrBkyRJqa2vJy8vjm9/8JieffDJHHXVUm+zS27dv59e//jWffvop06ZN48Ybb+Swww7br8yGhgZ27drFrl27sNlspKWltbkkJycPy643q7sqKSkpNCenfcDqbr2pgWINTkhLS8Nut+Pz+fB6vaG/1pLy0poJ3hqaH758faRJfwdKeLdiLGiAUgPC5/NRUVHRL79+A4EA77zzDitXrmTXrl3s3r27Tf95QkICEydO5H//93+ZNm1axOXW1dWxbt06NmzYwHvvvYfb7SYhIYHjjjuOb37zm2zfvp2XXnqJ1NRUFi9ezOzZs4dlkOmN7qYRhDPGUFlZOSiW97C62FJSUkhJSek2RVf7HImqLQ1Qql9ZM/sjmU/RVx6Ph5UrV/LCCy/w1VdfkZ2dzeGHHx7KCm39jUb3REtLC5s2bWLDhg1s2LCBsrIyRITvfOc7zJ8/f0ATfvami2mg9iUiJCUlkZmZ2eP1iioqKmhoaOi3IBX+Wm02G/Hx8V1moFc9pwFK9QsrWWpFRUXMl3eoq6vjL3/5C3/605+orKxk0qRJXHrppZxxxhn9siKuMYYdO3bgdDoZP358zPfXGRHB6XSSlZVFc3MzNTU1UX3fw7MRxMfHh5YxCQQClJWVdblkSUdluVwusrKyep0YOJJ5Oj3VPgg5HA6cTidOp7NNd5y2jGMj0gA1PNa5VjHh8XgoLy+PaU666uqe17q7AAAgAElEQVRqPvnkE95//31ee+01mpubOf744/nhD3/I0Ucf3a+/aEVkQAcxWCO6srOzSUxMDB38U1NTqa6u7lNXmPU+WiPJXC4XDodjv/d3zJgxlJeXd9uFKyLExcWRnZ3d5/kz1lBna0mSSJciCWd1ucXFxeF0OomPjw8FJJ1YPHhpgFLU1dVht9tJSkqK+DkNDQ1t1ryJBmMMJSUlfPzxx3zyySd8/PHH7Nq1CwiOeDr99NP54Q9/yMSJE6O2z6HCGvKblpa23wHYbreTnZ1NWloaVVVVEWd4Dw9KKSkpEXVh2e128vPzqa+vD7Wa25dps9nIysqKatohESE3N5eysrJQcAxvBYW3gKyWj91ux263h65r99zQowFqmLPSEEEwq3NGRka3X+TepC7qirXc9OOPP87u3buBYAqf6dOnM3v2bI488kgOP/zwPq0dNRRZ/4ekpCSysrK6/aUfFxdHXl4eLS0tVFdXh9IQhXe9Wn+tlldvz6tYAa2srCzUgu4qiEaDtK5DVFVVFermtLrkNPgcmDRADWP19fVtAo21hERubm6Hfe/GGKqrq6M6EOL9999nyZIlbN26lUmTJnH99dczY8YMDj744GHX9RI+aTMpKYmkpKROl7DvSnx8fL8sG+FwOBg5ciS1tbX4/X7S09Nj/j8TEbKysmK6DzV4aIAapsIHNlisRfj27t3LiBEj2gw8iPZoqq1bt/Lwww/z73//mxEjRnDHHXcwa9asfj8pHc2lM6yyelqmNRghOTk5lIl6qBCRmGQaUAo0QA1LXSVdNcbg9XopKioiPz8fl8sVSiQZjblN4QvQpaWlcd111/Gd73ynzSTYWBMREhMTycrKwuPxhJau6MtrExEyMzNDixnW1NQAXQ/LtgJTdnb2sOu+VCoSGqCGGbfbTWlpabcH40AgQElJCZmZmTQ2NtLS0tLn4FRYWMj8+fNpamri8ssv57LLLiMlJaVPZfaElQEgJycnNLLM4XCQmJiI2+2muroat9vd49aPde7HavlY2ddramo6zIQQzRFuSh3INEANIy0tLZSUlER8ALbSzVjX+2L79u3Mnz8fp9PJCy+80O/ziKwWTmpqaofndFwuFyNGjMDj8VBdXU1jYyM2m63LOT/djazLysoiLS0ttBR5+P3WMHGlVOc0QA0h1qTYurq60HLOTqez2wOdz+fD7XZ3OCw4kn321datW1mwYAEul4tHH32UMWPG9LnMSFkZDCIZBQfBJa3z8vLw+/14PB48Hg9utxuPxxPK8AzBlldeXl63XXNWiy09PR2Px6OBSake0AA1RDQ3N1NRUREaOux2u6mtrQWC81isE+wiEgpITU1NuN1u/H5/VAcD9MSWLVtYsGABSUlJPProox1mEo9ETwcgWN1oOTk5vTq/ZbfbSUhIICEhgbS0NCAYrK1VZHs6us7KTqCUipwGqEGupaWFioqKDrM1WLcbGxtDkzPbZ4Tu6Hp/+fzzz1m4cCGpqak8+uijjBw5slflWIMJcnNzqa+vDw1z7+g1xWqiqFW2Bhql+o8GqEHK6/VSWVkZ8ci59hMxB9pnn33GokWLSE9P59FHH2XEiBG9KscKTtaKsNYCao2NjVRVVeH3+9ssDZCenk5aWprmUFPqAKABahDy+XwUFRUNmmDTkdWrV7N69WpcLleoK8y62Gw2nn76abKysli6dGmvJ422D07h9ycnJ5OUlITb7aampga73d7jTNlKqcFNv82DULSzU0dbeXk5d911Vyjbgdvtprm5maamJnw+HwATJkzgkUceITc3t1f76Cw4td/GCopKqQNPTAOUiMwCHgTswB+MMfe0e3ws8AyQ3rrNTcaYlbGs02AXCASor68f6Gp06ZFHHsHn8/HEE0/sNyLP5/PR3NxMYmJir9PeRBKclFIHvph9+0XEDjwCnA1MBuaKyOR2m/0CeMkYcyTwXeD3sarPUFFXVzfQVejS559/zj/+8Q++973vdThc3OFwkJKSosFJKdVnsTwCHAMUGmN2GmM8wHJgTrttDGAtSZoGFMewPoOeMWZAuvcaGxsj2i4QCPDb3/6WrKws5s2bF5O6xMXFaXBSSgGxDVCjgD1ht4ta7wt3G/ADESkCVgKLOipIRK4WkU0isqm8vDwWdR0UorliaKSWL1/OaaedxmuvvdbttitXruTzzz9n0aJFPVo7KlLWmj8anJRSEGGAEpGTROSK1us5IjIhkqd1cF/7o+9c4GljzGjgHOA5EdmvTsaYx40xBcaYgpycnEiqPORYaYX6M0AVFRWxZMkSHA4Ht99+O6tXr+5028bGRh5++GGmTJnCOeecE/W6iAgpKSmaNFUpFdJtgBKRW4GfATe33hUHPB9B2UVA+EmK0ezfhXcl8BKAMeY9wAVkR1D2AcfK+NBfjDHcddddOBwOXnzxRY488khuueUW3njjjQ63f/rpp6moqOCGG26ISQvHypWnlFKWSI40FwDnA40AxphiIJIU1BuBiSIyQUScBAdBrGi3zW7gdAAROZxggDpw+/C60N+tp1dffZWNGzeyePFixo0bx/3338/06dP5v//7P95888022xYVFfH8889z7rnnMnXq1KjXRUTIzs7Wrj2lVBuRHBE8JnjkNAAiEtHJB2OMD1gIvA58QXC03mYRuUNEzm/d7KfAVSLyKbAMuNwM5glAMdLS0oLH4+m3/e3bt4/777+fo446igsuuAAI5vN74IEHmDJlCj//+c9Zt25daPsHHngAh8PBwoULY1KfuLi4mJzTUkoNbZHMg3pJRB4D0kXkKmAe8EQkhbfOaVrZ7r5bwq5vAU6MvLoHpv4cuWeM4Z577sHn8/GLX/yiTaslKSmJhx56iIULF3LTTTfxm9/8BqfTybp161iwYAGxOP8nIuTk5GiGb6XUfroNUMaY34rImUAdcChwizFmTcxrNkz4fD6ampr6bX9r165l/fr1/OQnP+lwHlNycjIPP/wwP/7xj7nxxhvJzMxk1KhRfO9734tJfZKSkvp1NV2l1NDRZRefiNhFZK0xZo0x5gZjzPUanKKrP1tPNTU13HvvvUyePJm5c+d2up0VpA4++GDKysq47rrrYhJERISsrKyol6uUOjB02YIyxvhFpElE0owxtf1VqQNJIBBARDrswurvtEb33XcfdXV1LF26tNukqqmpqSxdupTNmzdz7LHHdrhNX9aYskbt9TbjhFLqwBfJOSg38B8RWUPrSD4AY8zimNXqAOF2uyktLSUQCOBwOHC5XLhcLuLj43E6nf2a1ujtt99m1apVXHXVVRxyyCERPSclJYXjjjuu08et81eBQKDHgcrhcJCamtr9hkqpYSuSAPVa60X1QH19fZsl1n0+Hw0NDaG0QtHu1jPG8M4771BUVBTaj/W3sbGRzz//nIMOOogrrrgiKvuz1l5KSUmhsrKShoaGiF+TDoxQSkUikkESz7TOY5rUetc2Y4w3ttUauowxVFdXh1Z97ejxaHO73dx1112sWrUqdF98fHxozaSkpCQmT57M4sWLo5qpITk5GZvNRk5ODsnJyZSVlXW50i0QWkLd5XJFrR5KqQNTtwFKRE4luCTGLoLpi8aIyGXGmPWxrdrQY4xh3759oeXX+0NxcTHXX389O3bsYP78+Vx00UUkJSXFfFny9stpJCQkMHbsWCoqKtrkFBSRUHdecnKynnNSSkUski6++4CZxphtACIyieCk2qNiWbGhxu/3U1JSgtfr7bfg9P777/Pzn/8cv9/P/fffz0knndQv+xUR0tLS9rvfZrORm5tLc3MzlZWVuFwuUlNTNb+eUqpXIglQcVZwAjDGbBeR2P48H2K8Xi/FxcX9lkvPGMNzzz3Hww8/zPjx47nvvvs6nNMUKzabrcth5wkJCYwePbrf6qOUOjBFEqA2icgfgedab38f+DB2VRp6ysrK+i04NTc3c8cdd7BmzRpOP/10br31VhITE/tl3/Df1pMOcFBKxVokAWo+sABYTPAc1Hp05dsQr9eL1xvdMSObNm1izZo11NXVUV9fT11dHQ0NDaHbxhgWLVrEpZde2u+BwhhDSkokuYKVUqpvIglQDuBBY8zvILSUu+amaVVXVxfVc061tbX89Kc/BSArK4uUlBTS0tIYPXo0qamppKSkcMIJJ3DkkUdGbZ89mXCbkJCgAx2UUv0ikgD1JnAG0NB6OwF4AzghVpUaKowxUc8E8fzzz9PU1MSyZcsinlDbVzabDbvd3m1GdWvuk1JK9YdIlttwGWOs4ETr9f476TGIud3uqLaeqqqqWLZsGTNnzuy34CQiZGRkkJ+f3+16TCKi85eUUv0mkgDVKCIzrBsichTQHLsqDR2dTcbtraeffhqPx8PVV18dtTK7Yy217nA4GDFiRKfntHRwhFKqv0XSxfe/wJ9FxFqufQRwSeyqNDQEAgGam6MXp/ft28fLL7/Mueeey7hx46JWbleshK1W0ImPjycnJ4fy8vL9Aq8OjlBK9bdIUh1tFJHDCK4FJcBWTXUEDQ0N3W/UA0899RQ+n48f/ehHUS23K1brKVxycjItLS37Df5ISEjoNgO6UkpFU6ddfCJytIjkA7QGpBnAncB9IpLZT/UbtKI5eq+4uJhXXnmFb3/724waNSoqZXbHOvfUUZddZmYmLpcr9FhnmSOUUiqWujoH9RjgARCRk4F7gGeBWuDx2Fdt8Ir23Kc//OEP2Gw25s2bF7UyuyMinS53ISLk5eWFhpOLCAkJCf1WN6WUgq4DlN0YU9V6/RLgcWPMy8aY/wP6Z4jZIBXN1tPu3bt57bXXuOiii8jLy4tKmd3pqvVksdlsjBw5MhTIdHCEUqq/dRmgRMQ66XA68FbYYxGdjBCRWSKyTUQKReSmDh6/X0Q+ab1sF5GayKs+MKI99+mJJ54gLi6Oyy67LGpldqer1lM4h8PBmDFjdO6TUmpAdBVolgH/EpEKgsPKNwCIyCEEu/m61Jpx4hHgTKAI2CgiK4wxW6xtjDHXhW2/CIheeoQYiebcpy+//JLVq1dz6aWXkp2dHZUyuxNJ6ymcDoxQSg2UTo8+xpi7RORNgsPK3zD/PSrbgEURlH0MUGiM2QkgIsuBOcCWTrafC9waacUHSjS79x577DESExP54Q9/GJXyIhFp60kppQZalz+PjTH/7uC+7RGWPQrYE3a7CDi2ow1FZBwwgbbdiOGPXw1cDTB27NgIdx99gUCApqamqJS1detW3nrrLa666qp+60LraetJKaUGUiSZJHqro6NgZ02P7wJ/McZ0uGaFMeZxY0yBMaYgJycnahXsqcbGxqiUU1RUxG233UZqairf//73o1ImBANQfHw88fHxbUbghV90sq1SaqiI5QmGIiB8Fb3RQHEn236X4JIeg1o0Uhu99dZb3H777dhsNu666y6Sk5OjVLugESNGhHLqGWPw+/34fD68Xi9Op7PbfHtKKTVYdBugRGQh8IIxprqHZW8EJorIBGAvwSD0vQ7KPxTIAN7rYfn9qq9zn7xeLw899BDLli1jypQp3H333YwcOTJq9bMyjYcHIBHB4XDgcDg0yatSasiJpAWVT3AE3kfAk8DrJoJmhDHG1xrcXgfswJPGmM0icgewyRizonXTucDySMocSDU1Nb1uPZWUlHDTTTexefNm5s6dy+LFi4mLi4tq/TTbg1LqQCORHHQleFZ9JnAFUAC8BPzRGPNlbKu3v4KCArNp06Z+3WdLSwvFxcW9ClAbNmzg1ltvxe/3c8stt3D66adHvX5W0lcNUEqpoUBEPjTGFHS3XUQnJFpbN6WtFx/BLrm/iMi9farlEGCM6TC7d3d8Ph9LlizhuuuuY8SIEbzwwgsxCU4QzPqgQ8eVUgeaSM5BLQYuAyqAPwA3GGO8ImIDdgA3xraKA6uxsbHH554qKyv5+c9/zocffsgFF1zA9ddfT3x8fEzqJyJkZWXp0HGl1AEnknNQ2cCFxpivw+80xgREZHZsqjU4BAIBKioqetR6+uSTT7jpppuor6/ntttuY/bs2L5FDoeDpKSkmO5DKaUGQiRdfCsBK2ksIpIiIscCGGO+iFXFBoPq6uqIg5MxhhdffJFrrrkGl8vFU089FfPgpK0npdSBLJIAtRQIX52vsfW+A5rX6404rVFjYyM333wzv/vd7zjppJN47rnnmDRpUszr6HQ6dRkMpdQBK5IuPgkfAt7atXfAZxCNdGBEaWkpCxYsYM+ePSxatIhLL700Ki0ah8NBIBDAGNNhPbT1pJQ60EUSaHa2DpSwWk0/BnbGrkoDr6mpiZaWloi2feqppygtLeX3v/89BQXdjpqMiIiQn59PXFwczc3N1NTU4Ha722zjcrl08q1S6oAWSRfftcAJBLNBWAlfr45lpQZST4aVu91uVq9ezWmnnRbV4JSSkoLT6URESExMZOTIkYwdO5a0tLRQpoisrKyo7E8ppQarbltQxph9BNMUDQs1NTUEAoGItv3nP/9JY2Mjc+bMidr+rUm37TkcDrKyssjMzAzl1VNKqQNZJPOgXMCVwBQg1KdkjJkXw3oNCJ/P16OURitWrGDUqFHMmDEjKvu3glNXCV1FRIOTUmpYiKSL7zmC+fjOAv5FMCt59NY8H0R6stbT3r172bhxI+edd17UMoTHxcXpchhKKdUqkiPrIcaY/wMajTHPAOcCR8S2WgPD5/NF3Hr6xz/+gYhEba6TiJCTk6Oj8pRSqlUkAcrK81MjIlOBNGB8zGo0gCJNaeT3+1mxYgXHHXcc+fn5Udl3UlJSzNIhKaXUUBRJgHpcRDKAXwArgC3Ar2NaqwHi8/ki2m7jxo2UlZVx/vnnR2W/1pwmpZRS/9XlIInWhLB1rYsVrgcO6pdaDRC/v8MV5/fz6quvkpaWximnnNLnfVoDI6wl2pVSSgV12YIyxgSAhf1UlwEXSYCqra1l3bp1zJo1Kyqj6RwOhy6VoZRSHYiki2+NiFwvImNEJNO6xLxm/ayzlELtrV69Gq/XG5XuPR0YoZRSnYsk1ZE132lB2H2GA6y7z+fzISLdBqkVK1Zw2GGHceihh/Z5nwkJCZquSCmlOhFJJokJ/VGRgRZJ997WrVvZtm0bN97Y9zUaRYTs7Ow+l6OUUgeqSDJJXNrR/caYZ6NfnYETyQi+v//97zidTs4666w+7UtESE9Px+E44JPCK6VUr0VyDurosMs3gduAiE7AiMgsEdkmIoUiclMn2/yPiGwRkc0i8mKE9Y46v9/fZfdeS0sLq1at4tRTTyUtLa1P+7LZbKSnp/epDKWUOtBF0sW3KPy2iKQRTH/UJRGxA48AZxLMgr5RRFYYY7aEbTMRuBk40RhTLSK5Pax/1HQ3SXf9+vXU1dX1eXCE1bWnAyOUUqprvUki1wRMjGC7Y4BCY8xOY4wHWA60T/t9FfBI6zwrK3P6gOiui+/VV18lPz+fY445pk/7cTqdJCYm9qkMpZQaDiI5B/V3gqP2IBjQJgMvRVD2KGBP2G1rLalwk1r38Q5gB24zxqyOoOyo6ypAvfvuu7z//vv86Ec/6lNiWB1WrpRSkYvkLP1vw677gK+NMUURPK+jo3D7kzwOgq2xUwlmSd8gIlONMTVtChK5mtZFEseOHRvBrnuus1F8f/vb37j77ruZOHEil1xySa/LD1+IUCmlVPciCVC7gRJjjBtARBJEZLwxZlc3zysCxoTdHg0Ud7DNv40xXuArEdlGMGBtDN/IGPM48DhAQUFBZOnGe8AYs1+AMsawdOlSnnzySU444QTuvvtukpKS+rSfjIyMPj1fKaWGk0j6q/4MhC8x62+9rzsbgYkiMkFEnARX5V3Rbpu/Ad8CEJFsgl1+OyMoO6rar6Dr9Xq55ZZbePLJJ/n2t7/N7373uz4FJysZrObbU0qpyEXSgnK0DnIAwBjjaQ04XTLG+ERkIfA6wfNLTxpjNovIHcAmY8yK1sdmisgWgoHvBmNMZa9eSR/4/f5QFon6+npuuOEGNm3axPz585k3b16fzxk5HA5diFAppXookgBVLiLntwYURGQOUBFJ4caYlcDKdvfdEnbdAP9f62XAWGmOSkpKWLx4Mbt37+aOO+7gnHPO6XPZOjBCKaV6J5IAdS3wgog83Hq7COgwu8RQ5ff7aWpq4sorr6SxsZElS5Zw9NFH96lMEcHpdJKbm0tcXFyUaqqUUsNHJBN1vwSOE5FkQIwx9bGvVv/y+Xzs2rWLsrIy7rzzzqgEp4yMDNLS0rTlpJRSvdTtIAkR+ZWIpBtjGowx9SKSISJ39kfl+ovX66WsrAyAMWPGdLN150QEh8PBqFGjSE9P1+CklFJ9EMkovrPD5yW1Zn3o+8mZQcTn81FaWgpAfn5+r8oQEVJTUxkzZozOdVJKqSiI5ByUXUTijTEtEJwHBcTHtlr9y+/3U1ZWRlxcXK/mKokIeXl5msJIKaWiKJIA9Tzwpog8RTATxDzggFpqwwpQubm5vUplZLfbSUhIiEHNlFJq+IpkkMS9IvIZcAbB9EW/NMa8HvOa9RNjDIFAgNLS0l5174mIDoZQSqkYiKi5YIxZbYy53hjzU6BBRB6Jcb36jTVJt6ysjLy8vF6VkZycHOVaKaWUimhJVxH5BjAXuAT4CvhrLCvVn/x+P36/n/Ly8l4FqMTERE1hpJRSMdBpgBKRSQTz580FKoE/EZwH9a1+qlu/8Pl8VFRU4Pf7e9zFZ43cU0opFX1dtaC2AhuA84wxhQAicl2/1Kof+Xy+0ByonragbDYbLpcrFtVSSqlhr6tzUBcBpcA/ReQJETmdjtd4GtJ6OwdKB0copVRsdRqgjDGvGGMuAQ4D1gHXAXkislREZvZT/WKutwHKGKMZypVSKoa6HcVnjGk0xrxgjJlNcNHBT4CbYl6zfmJ18SUlJfVoNF5CQoIOjlBKqRjq0axUY0yVMeYxY8xpsapQf7NaUD05/2R17ymllIqdnqdNOMAEAgHKysp6fP5JM0copVRsDesAFQgEMMb0eJKuDo5QSqnYG9YByu/34/F4qKqq6lGA0sERSikVe8M6QPl8Pvbt2wdEPoLP5XLhcESUgEMppVQfDOsA5ff7Q0PMI2lB6eAIpZTqP8M6QIXPgYo0QOmaT0op1T9iGqBEZJaIbBORQhHZb+6UiFwuIuUi8knr5UexrE974QEqNze3fd2w2WzY7Xbi4uJwOp1kZWXp4AillOonMTuZIiJ24BHgTKAI2CgiK4wxW9pt+idjzMJY1aMrXq+XsrIyMjIyQjn10tLSyMzM1ECklFIDLJYtqGOAQmPMTmOMB1gOzInh/nrMyiJhde+JCHFxcRqclFJqEIhlgBoF7Am7XdR6X3sXichnIvIXERnTUUEicrWIbBKRTeXl5VGroDVIwhrBJyI6Qk8ppQaJWAaojpohpt3tvwPjjTHTgLXAMx0VZIx53BhTYIwpyMnJiUrlrKXew1tQxhji4uKiUr5SSqm+iWWAKgLCW0SjgeLwDYwxlcaYltabTwBHxbA+bQQCARoaGmhsbGwToLQFpZRSg0MsA9RGYKKITBARJ8HVeVeEbyAiI8Jung98EcP6tOH3+0MLFVpdfDabTc8/KaXUIBGz5oIxxiciC4HXATvwpDFms4jcAWwyxqwAFovI+YAPqAIuj1V92utoJV1tPSml1OAR0yOyMWYlsLLdfbeEXb8ZuDmWdehMR5N0NUAppdTgMWwzSVgBym63k52dDYDT6RzgWimllLIM6wBVVlZGdnY2DocjNAdKKaXU4DCsA1T7lXS1i08ppQaPYR2g2q+kqy0opZQaPIZtgPJ6vezbt6/NHCi73T7AtVJKKWUZlgHKGENVVRUejycUoHQOlFJKDS7DMkD5/f79VtLV809KKTW4DMsAFT4HygpQev5JKaUGl2EZoDpa6l3nQCml1OAyLAOUNYIvPj6e9PR0nQOllFKD0LAOUHl5eaGBEXoOSimlBpdhGaC8Xq9O0lVKqUFuWAao9pN0dQ6UUkoNPsMyQLndbsrLy0MtKLvdrnOglFJqkBmWAWrfvn0YY3SZDaWUGsSGXYAKBAI6B0oppYaAYdd0CA9QOgdKqY55vV6Kiopwu90DXRU1hLlcLkaPHt3rRsCwC1BAm6XeRUS7+JRqp6ioiJSUFMaPH6/nZ1WvGGOorKykqKiICRMm9KqMYdfFB8EAlZKSQlJSEqBdfEq153a7ycrK0uCkek1EyMrK6lMrfFgGqPA5UMYYbUEp1QENTqqv+voZimmAEpFZIrJNRApF5KYutvuOiBgRKYhlfSxWFonWfWOzDcs4rZRSg1rMjswiYgceAc4GJgNzRWRyB9ulAIuB92NVl/ZKS0tDI/h0DpRSg09lZSXf+MY3+MY3vkF+fj6jRo0K3fZ4PBGVccUVV7Bt27Yut3nkkUd44YUXolFlFQOx7Ns6Big0xuwEEJHlwBxgS7vtfgncC1wfw7qENDU1UVtbq3OglBrEsrKy+OSTTwC47bbbSE5O5vrr2x4ijDEYYzrtAXnqqae63c+CBQv6XtkY6O61DRexfPWjgD1ht4ta7wsRkSOBMcaYf3RVkIhcLSKbRGRTeXl5nypVXFwM6BBzpXrivPPO2+/yxz/+EQj+6Ovo8RdffBEItobaP9ZbhYWFTJ06lWuvvZYZM2ZQUlLC1VdfTUFBAVOmTOGOO+4IbXvSSSfxySef4PP5SE9P56abbmL69Okcf/zxoQVLf/GLX/DAAw+Etr/ppps45phjOPTQQ3n33XcBaGxs5KKLLmL69OnMnTuXgoKCUPAMd8MNNzB58mSmTZvGz372MyDYWzNnzhymTZvG9OnTef/9YEfRvffey9SpU5k6dSpLlizp9LWtWrWK448/nhkzZnDJJZfQ2NjY6/duKIplgOqo38yEHhSxAfcDP+2uIGPM48aYAmNMQU5OTp8qtXfvXkAn6So1VG3ZsoUrr7ySjz/+mFGjRnHPPep2atIAABYvSURBVPewadMmPv30U9asWcOWLe07aaC2tpZTTjmFTz/9lOOPP54nn3yyw7KNMXzwwQf85je/CQW7JUuWkJ+fz6effspNN93Exx9/vN/zysrKWLlyJZs3b+azzz7j5ptvBoIttDPPPJPPPvuMDz/8kMMPP5wPPviAF154gQ8++ID33nuP3//+93z22Wf7vba4uDjuuece3nzzTT766COmTZvGgw8+GK23cUiIZf9WETAm7PZooDjsdgowFVjXeg4oH1ghIucbYzbFqlJWgNI5UEpF7u9//3unjyUmJnb5eFZWVpeP99TBBx/M0UcfHbq9bNky/vjHP+Lz+SguLmbLli1Mntz2dHdCQgJnn302AEcddRQbNmzosOwLL7wwtM2uXbsAePvtt0MtounTpzNlypT9npeZmYnNZuOqq67i3HPPZfbs2QCsW7eO5cuXA8HTCampqWzYsIGLLrqIxMREAL797W/z9ttvM3PmzDav7d1332XLli2ccMIJAHg8Hk466aSev2FDWCyPzhuBiSIyAdgLfBf4nvWgMaYWyLZui8g64PpYBif4b4DKzc0FtAWl1FBjzV8E2LFjBw8++CAffPAB6enp/OAHP+hw3k14V77dbsfn83VYdnx8/H7bGGM63DZcXFwcmzZtYs2aNSxfvpylS5fyxhtvAPsPte6qvPDXZoxh1qxZPPfcc93u/0AVsy4+Y4wPWAi8DnwBvGSM2Swid4jI+bHab3eKi4vJysrC6XTqHCilhri6ujpSUlJITU2lpKSE119/Per7OOmkk3jppZcA+M9//tNhF2J9fT11dXXMnj2b+++/P9QN+K1vfYtHH30UAL/fT11dHSeffDKvvPIKzc3NNDQ08Oqrr/LNb35zvzJPOOEE/vWvf7Fz504geC5sx44dUX99g1lMj87GmJXAynb33dLJtqfGsi6WoqKi0PknnQOl1NA2Y8YMJk+ezNSpUznooIM48cQTo76PRYsWcemllzJt2jRmzJjB1KlTSUtLa7NNbW0tF154IS0tLQQCAX73u98B8PDDD3PVVVfx2GOP4XA4eOyxxzjmmGOYO3duqCtv/vz5HHHEERQWFrYpMy8vjz/+8Y9ccskloaH1v/rVr5g4cWLUX+NgJZE0XweTgoICs2lT73sBjz32WMaMGcO9996Lw+Fg7NixUaydUgeGL774gsMPP3ygqzEo+Hw+fD4fLpeLHTt2MHPmTHbs2KG9LxHq6LMkIh8aY7pNzDDs3uFrrrkm1GrS809Kqe40NDRw+umn4/P5MMaEWkMq9obdu3zppZfy/7d3/tFRVFke/1xCIKAoQcbZMNFJYMDdOGGCRoggP2ScGKJHCKKAjIoOulnRE/SgwOJwFo4iMhwny48DCsK4jOIPMBFRlxXJgOPyMxggYhBQh1+ChixICDga7v5RL02n04mBSacbcj/n9EnVe6+qvnXTVbffq1f37tu3D1U1B2UYxo/Stm1bCgsLwy2jSdJkH8CIiL2kaxiGEcE0WQcFFubIMAwjkjEHZRiGYUQkTdZB2TtQhmEYkU2TdFBVUYLtHSjDiFwOHTrEsGHD6NSpE0lJSWRmZvLZZ5+FW1ZQEhISKC0tBfCFJgpk5MiRLF26tM79/OlPf/IFtAYYNWpU0BeDmwpN9g4dFRUVbgmGYdSCqpKVlUW/fv3Ys2cPO3bsYOrUqRw+fLhau8rKyjAprJ2qKOjnQqCDWrBgQY24gpFAbaGiGpomO8ZlU8wNo35MmDCB4uLiBt3nL3/5S5555pla6wsKCoiOjiY7O9tXlpKSAngBWCdPnkxcXBxFRUXs2LGD5557zhehfNSoUYwZM4YTJ05w5513sn//fiorK/n973/P0KFDGT9+PMuXL6d58+akp6czY8aMaseeO3cuX3zxBdOnTwc8p1FYWMisWbMYNGgQ+/bt49SpU+Tk5PDggw/W0H7xxRdTXl6OqvLII4+wevVqEhMTq8XgmzJlCm+//TYnT56kZ8+ePP/88yxbtozNmzczYsQIWrVqxbp16xgwYAAzZswgNTWVJUuWMHXqVFSVW265hWeffdZ3vJycHFasWEGrVq146623fOmEqlizZg05OTmAN4N57dq1tGnThunTp7N48WKaNWvGgAEDmDZtGkVFRWRnZ1NRUUGnTp1YuHAhsbGx9OvXj549e/LRRx9x2223cc8995Cdnc3evXsByM3NbfBIHuagDMOIOIqLi7n22mtrrd+4cSPFxcUkJiZSWFjIokWL2LBhA6pKjx496Nu3L59//jkdOnTgnXfeAbxwRGVlZeTl5VFSUoKIcPTo0Rr7HjJkCNdff73PQb322mtMnDgRgIULF9KuXTtOnjzJddddx+23385ll10WVGNeXh47d+5k+/btHD58mKSkJO6//34AHn74YSZN8qK+3X333axYsYIhQ4Ywe/Zsn0Py5+DBg4wbN47CwkJiY2NJT08nPz+fQYMGceLECdLS0nj66ad54oknmD9/Pk8++WS17WfMmMGcOXPo1asX5eXlxMTE8N5775Gfn8+GDRto3bo1ZWVlgPeu6KxZs+jbty+TJk1i8uTJvpxZR48eZc2aNQDcddddPProo9xwww3s3buXm2++mU8//bSO/+rZYw7KMIw6qaunEy66d+9OYmIi4KXDyMrK8kUCHzx4MB9++CEZGRmMHTuWcePGceutt9K7d29fyKJRo0ZVS4vhz09+8hM6duzI+vXr6dy5Mzt37vT1DGbOnEleXh4A+/btY9euXbU6qLVr1zJ8+HCioqLo0KED/fv399UVFBQwffp0KioqKCsr4+qrr64zkeOmTZvo168fVfnwRowYwdq1axk0aBAtWrTwnce1117L+++/X2P7Xr168dhjjzFixAgGDx5MfHw8q1at4r777vOl/WjXrh3Hjh3j6NGj9O3bF4B7772XO+64w7efoUOH+pZXrVpV7fnYt99+y/Hjx2nTpk2t53G2NNlnUOagDCNyufrqq+uM3hCYliIYXbp0obCwkOTkZCZMmMCUKVNo3rw5Gzdu5Pbbbyc/P5+MjAwqKytJSUkhJSXF16sZOnQor7/+OsuWLSMrKwsR4S9/+QurVq1i3bp1bN26lW7dugVN7eFPYKoNgFOnTvHQQw+xdOlStm/fzgMPPPCj+6krZmp0dLTvOLWlEhk/fjwLFizg5MmTpKWlUVJSgqoG1VcX/nY/ffo069ato6ioiKKiIg4cONCgzgmasIOyKeaGEbn079+f7777jvnz5/vKNm3a5Bte8qdPnz7k5+dTUVHBiRMnyMvLo3fv3hw8eJDWrVvz29/+lrFjx7JlyxbKy8s5duwYmZmZ5ObmUlRURFRUlO8mW5VFd/DgweTn57NkyRJfr+HYsWPExsbSunVrSkpKWL9+fZ3n0KdPH1599VUqKyv56quvKCgoAPA5o/bt21NeXl5tZl+bNm04fvx4jX316NGDNWvWUFpaSmVlJUuWLPH1curDnj17SE5OZty4caSmplJSUkJ6ejoLFy6koqICgLKyMi699FJiY2N9CR0XL15c63HS09OZPXu2b72oqKjeeupLk71Lm4MyjMhFRMjLy2PMmDFMmzaNmJgYEhISyM3N9SUdreKaa65h5MiRdO/eHfAmSXTr1o2VK1fy+OOP06xZM6Kjo5k7dy7Hjx9n4MCBnDp1ClXlj3/8Y9Djx8bGkpSUxI4dO3z7zcjIYN68eXTt2pWrrrqKtLS0Os8hKyuL1atXk5ycTJcuXXw3+rZt2/LAAw+QnJxMQkJCtezAI0eOJDs72zdJooq4uDieeeYZbrzxRlSVzMxMBg4cWG975ubmUlBQQFRUFElJSQwYMICWLVtSVFREamoqLVq0IDMzk6lTp/LSSy/5Jkl07NiRRYsWBd3nzJkzGT16NF27duWHH36gT58+vtxXDUWTS7dx+vRpSktLfRl1DcOoiaXbMBqKfyTdRpMb4mvWrJk5J8MwjPOAJuegDMMwjPMDc1CGYQTlfBv+NyKPf/Q7ZA7KMIwaxMTEcOTIEXNSxjmjqhw5coSYmJhz3kdIp7KJSAbwn0AUsEBVpwXUZwOjgUqgHHhQVZtuZETDiBDi4+PZv38/33zzTbilGOcxMTExxMfHn/P2IXNQIhIFzAF+A+wHNonI8gAH9IqqznPtbwOeAzJCpckwjPoRHR3ti9RgGOEilEN83YHdqvq5qv4deBWoNnFfVb/1W70IsPEEwzAMAwjtEN/PgH1+6/uBHoGNRGQ08BjQAugfWO/aPAg8CHDllVc2uFDDMAwj8ghlDypYkKcaPSRVnaOqnYBxwJM1NwFVfUFVU1U1tSpYomEYhnFhE8oe1H7gCr/1eOBgLW3BGwKc+2M7LSwsLBWRv52DnvZA6Tls19iYzobFdDYsprPhOV+0NqTOn9enUSgd1Cags4gkAgeAYcBd/g1EpLOq7nKrtwC7+BFU9Zy6UCKyuT6hNcKN6WxYTGfDYjobnvNFazh0hsxBqeoPIvIwsBJvmvlCVf1ERKYAm1V1OfCwiNwEfA/8H3BvqPQYhmEY5xchfQ9KVd8F3g0om+S3nBPK4xuGYRjnL00pksQL4RZQT0xnw2I6GxbT2fCcL1obXed5l27DMAzDaBo0pR6UYRiGcR5hDsowDMOISC54ByUiGSKyU0R2i8j4MGu5QkQKRORTEflERHJceTsReV9Edrm/sa5cRGSm075NRK5pZL1RIvKxiKxw64kissHpfE1EWrjylm59t6tPaESNbUVkqYiUOLteH4n2FJFH3f+8WESWiEhMpNhTRBaKyNciUuxXdtY2FJF7XftdItLgM3Jr0fkH97/fJiJ5ItLWr26C07lTRG72Kw/pPSGYTr+6sSKiItLerUeUPV35I84+n4jIdL/yxrenql6wH7zp7XuAjnihlLYCSWHUEwdc45bbAJ8BScB0YLwrHw8865YzgffwonKkARsaWe9jwCvACrf+OjDMLc8D/s0tPwTMc8vDgNcaUeNLwCi33AJoG2n2xAv79QXQys+OIyPFnkAf4Bqg2K/srGwItAM+d39j3XJsI+hMB5q75Wf9dCa5670lkOjuA1GNcU8IptOVX4H32s3fgPYRas8bgVVAS7d+eTjtGfKLM5wf4Hpgpd/6BGBCuHX56XkLL9r7TiDOlcUBO93y88Bwv/a+do2gLR74AC8+4gp3AZX63Qx8tnUX3fVuublrJ42g8RK8G78ElEeUPTkTl7Kds88K4OZIsieQEHCjOisbAsOB5/3Kq7ULlc6AuizgZbdc7Vqvsmlj3ROC6QSWAr8CvuSMg4ooe+L9aLopSLuw2PNCH+ILFrD2Z2HSUg03bNMN2AD8VFW/AnB/L3fNwqk/F3gCOO3WLwOOquoPQbT4dLr6Y659qOkIfAMsckORC0TkIiLMnqp6AJgB7AW+wrNPIZFnT3/O1oaRcK3dj9cboQ49YdEpXjqhA6q6NaAqonQCXYDebmh5jYhcF06dF7qDqlfA2sZGRC4GlgFjtHrKkRpNg5SFXL+I3Ap8raqF9dQSLjs3xxuimKuq3YATeMNRtREue8bipZpJBDrgpZYZUIeWiPzeOmrTFlbNIjIR+AF4uaqoFj2NrlNEWgMTgUnBqmvRE85rKhZvuPFx4HURkTr0hFTnhe6gzjZgbcgRkWg85/Syqr7pig+LSJyrjwO+duXh0t8LuE1EvsQL4tsfr0fVVkSqoo/4a/HpdPWXAmWNoHM/sF9VN7j1pXgOK9LseRPwhap+o6rfA28CPYk8e/pztjYM27XmJhDcCoxQN84UYTo74f042equqXhgi4j8U4TpxB33TfXYiDeC0j5cOi90B+ULWOtmSA0DlodLjPsl8iLwqao+51e1nDNxCO/FezZVVX6Pm+mTBhyrGnYJJao6QVXjVTUBz2arVXUEUAAMqUVnlf4hrn3If+2p6iFgn4hc5Yp+DewgwuyJN7SXJiKt3XegSmdE2TOAs7XhSiBdRGJdjzHdlYUUEcnAS9Vzm6pWBOgfJt6MyESgM7CRMNwTVHW7ql6uqgnumtqPN1nqEBFmTyAfl5dPRLrgTXwoJVz2bOiHbpH2wZsl8xneTJOJYdZyA173dxtQ5D6ZeM8XPsCL5v4B0M61F2CO074dSA2D5n6cmcXX0X0pdwNvcGamT4xb3+3qOzaivhRgs7NpPt7wRMTZE5gMlADFwGK82VARYU9gCd6zse/xbp6/Oxcb4j0D2u0+9zWSzt14z0Cqrqd5fu0nOp07gQF+5SG9JwTTGVD/JWcmSUSaPVsAf3bf0y1A/3Da00IdGYZhGBHJhT7EZxiGYZynmIMyDMMwIhJzUIZhGEZEYg7KMAzDiEjMQRmGYRgRiTkoo8kgIpeJSJH7HBKRA37rLeq5j0V+713V1ma0iIxoIM0Dnb6tIrJDREb9SPv+7n2aYHVxIvKu376Wu/IrROS1htBrGA2JTTM3miQi8h9AuarOCCgXvOvidNANGxERaYkXDDdVVQ+69Z+r6md1bPMUUKqquUHqXgS2qOoct95VVbeFSL5h/MNYD8po8ojIL8TL0zQP7+XEOBF5QUQ2u5w4k/za/lVEUkSkuYgcFZFprkeyTkQud22eEpExfu2nichG8XLm9HTlF4nIMrftEneslABpl+K9yFkGoKrfVTknEfmpiLzpttsoImki0gkYBTzuel09A/YXh/dCJm5/2/zOv8gtL/LrVZaKF+MOERnvjrPN3x6GEUrMQRmGRxLwoqp2Uy/6+HhVTcVLj/AbEUkKss2lwBpV/RWwDu/N/2CIqnbHC75ZdXN/BDjktp2GF9m+Gqr6NS5/kIi8IiLDRaTqmp0JTHca7wQWqOoeYAHwB1VNUdX/DdjlbOAlEVktIv8uLtZewDHvU9UUvNQVpcB/iUgmcCXQAy9yR88gzs8wGhxzUIbhsUdVN/mtDxeRLXg9qn/Bc2CBnFTVqvQOhXi5dYLxZpA2N+AF4kW9FAyfBNtQVUfi5QzbjBep/QVXdRMwz/V88oFYEWlV++mBqr6LF7j0RXc+H4tIjTQebj9v4CVQ3IcXB24A8DGePX6Bl5bBMEJK8x9vYhhNghNVCyLSGcgBuqvqURH5M158vED+7rdcSe3X03dB2gRLUxAUNxS3TUReAT7FG8YTp89fA94jtDr3dQQvJcXLIvLfeI4y0DnOB15V1QI/rU+p6ov11WwYDYH1oAyjJpcAx4Fv3TDYzSE4xl/xhuYQkWSC9NBE5BIR6eNXlIKXLhy8tNyj/dpWPb86DrQJdkAR+XVVL0tELsFLAbE3oE0OEB0weWQl8DvxkkEiIvEi0r6e52kY54z1oAyjJlvw0mEUA58DH4XgGLPwnu9sc8crxsuc648AE0RkPnASKOfMc67RwFwRuQ/vOi5wZW8Bb4jIYGB0wHOo64DZIvI93o/Tuar6sYj8wq/NWKCiatIEMFtVF4jIPwPrXQ/tOHAX3jMqwwgZNs3cMMKAeIkIm6vqKTek+D9AZz2TAt4wmjzWgzKM8HAx8IFzVAL8qzknw6iO9aAMwzCMiMQmSRiGYRgRiTkowzAMIyIxB2UYhmFEJOagDMMwjIjEHJRhGIYRkfw/1Hv7tqtQjSwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.datasets import load_digits\n",
    "from sklearn.model_selection import learning_curve\n",
    "\n",
    "# load data\n",
    "digits = load_digits()\n",
    "\n",
    "# create feature matrix and target vector\n",
    "features, target = digits.data, digits.target\n",
    "\n",
    "# reate CV training and test scores for various training set sizes\n",
    "train_sizes, train_scores, test_scores = learning_curve(RandomForestClassifier(), \n",
    "                                                        features, \n",
    "                                                        target, \n",
    "                                                        cv=10, \n",
    "                                                        scoring='accuracy', \n",
    "                                                        n_jobs=-1, \n",
    "                                                        train_sizes=np.linspace(0.01, 1.0, 50))\n",
    "\n",
    "# create means and standard deviations of training set scores\n",
    "train_mean = np.mean(train_scores, axis=1)\n",
    "train_std = np.std(train_scores, axis=1)\n",
    "\n",
    "# create means nad standard deviations of test set scores\n",
    "test_mean = np.mean(test_scores, axis=1)\n",
    "test_std = np.std(test_scores, axis=1)\n",
    "\n",
    "# draw lines\n",
    "plt.plot(train_sizes, train_mean, '--', color=\"#111111\", label=\"Training score\")\n",
    "plt.plot(train_sizes, test_mean, color=\"#111111\", label=\"Cross-validation score\")\n",
    "\n",
    "# draw bands\n",
    "plt.fill_between(train_sizes, train_mean - train_std, train_mean + train_std, color=\"#DDDDDD\")\n",
    "plt.fill_between(train_sizes, test_mean - test_std, test_mean + test_std, color=\"#DDDDDD\")\n",
    "\n",
    "# create plot\n",
    "plt.title(\"Learning Curve\")\n",
    "plt.xlabel(\"Training Set Size\"), plt.ylabel(\"Accuracy Score\")\n",
    "plt.legend(loc=\"best\")\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Discussion\n",
    "Learning curves visualize the performance (e.g. accuracy, recall) of a model on the training set and during cross-validation as the number of observatino in the training set increases. They are commonly used to detemrine if our learning algorithms would benefit from gathering addition training data.\n",
    "\n",
    "In our solution, we plot the accuracy of a random forest classifier at 50 different training set sizes ranging from 1% of observation to 100%. THe increasing accuracy score of the cross-validated models tell us that we would likely benefit from additional observations (although in practice this might not be feasible).\n",
    "\n",
    "#### See Also\n",
    "* scikit-learn documentation: Learning Curve (http://scikit-learn.org/stable/modules/learning_curve.html#learning-curve)\n",
    "\n",
    "### 11.12 Creating a Text Report of Evaluation Metrics\n",
    "#### Problem\n",
    "You want a quick description of a classifier's performance.\n",
    "\n",
    "#### Solution\n",
    "Use scikit-learn's `classification_report`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "     setosa       1.00      1.00      1.00        13\n",
      " versicolor       1.00      0.62      0.77        16\n",
      "  virginica       0.60      1.00      0.75         9\n",
      "\n",
      "avg / total       0.91      0.84      0.84        38\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn import datasets\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import classification_report\n",
    "\n",
    "# load data\n",
    "iris = datasets.load_iris()\n",
    "\n",
    "# create feature matrix\n",
    "features = iris.data\n",
    "\n",
    "# create target vector\n",
    "target = iris.target\n",
    "\n",
    "# create list of target class names\n",
    "class_names = iris.target_names\n",
    "\n",
    "# create training and test set\n",
    "features_train, features_test, target_train, target_test = train_test_split(features, target, random_state=1)\n",
    "\n",
    "# create logistic regression\n",
    "classifier = LogisticRegression()\n",
    "\n",
    "# train model and make predictions\n",
    "model = classifier.fit(features_train, target_train)\n",
    "target_predicted = model.predict(features_test)\n",
    "\n",
    "# create classification report\n",
    "print(classification_report(target_test, target_predicted, target_names=class_names))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Discussion\n",
    "`classification_report` provides a quick means for us to see some common evaluation metrics, including precision, recall, and F1-score (described earlier in this chapter). Support refrs to the number of observations in each class\n",
    "\n",
    "#### See Also\n",
    "* Precision and recall (https://en.wikipedia.org/wiki/Precision_and_recall)\n",
    "\n",
    "### 11.13 Visualizing the Effect of Hyperparameter Values\n",
    "#### Problem\n",
    "You want to understand how the performance of a model changes as the value of some hyperparameter changes\n",
    "\n",
    "#### Solution\n",
    "Plot the validation curve:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzsvXl8nNV1//8+mhntqyVZiyV5wyy2AduY1TaYLYFAgLCEkH1pSL6/JG3yTdqmbZqmNEnTJN82bbM0pIUkQCCUhITFrDFgEowXDLaxwMYIY8mybO27Zr2/P56FZ0YjaWRrNAKd9+ull+bZz/PMM/dzz73nnivGGBRFURRlppGVaQMURVEUJRkqUIqiKMqMRAVKURRFmZGoQCmKoigzEhUoRVEUZUaiAqUoiqLMSFSglAkRkQUiYkTEby8/IiIfS2XfY7jW34rIfx+Pve90RKRBRAZExDfOPkZETphOu1JFRA6IyCWZtkOZ+ahAzQJE5DERuSXJ+qtFpG2yYmKMudwY84spsGu9iLQknPvbxpg/O95zj3G9GhH5HxE5LCL9IvKqiPyjiBSk43qTsOunIvJjz3JARAbHWHeOMeagMabQGBO1tz0tIsf8zETkGyIStkWvR0SeE5Fzj++uMo+I/FxEQvZ9OX83ZtouJXVUoGYHPwc+IiKSsP4jwF3GmMj0mzS9iMgcYDOQB5xrjCkCLgVKgcXHcL5j8hDHYBNwgWd5NXAQOD9hHcALU3hdL782xhQCFcBTwP+m6TrTzXdtMXf+fj3ZE4znqSrpRQVqdvA7YA6wzlkhImXAlcAv7eUrRORFEekTkWYR+cZYJ/PW2EXEJyLfF5EOEWkCrkjY9xMi8ortsTSJyGfs9QXAI0Ctp3Zba9fm7/Qcf5WI7LFr9k+LyCmebQdE5CsisktEekXk1yKSO4bZ/xfoBz5sjDkAYIxpNsb8hTFmV7KmyYT7/LiI/ElE/k1EuoB/sm1a7tm/UkSGRWSuvXyliLzk8UpOG8O2Z4BTRKTCXl4H3AMUJKzbbIwJe20VkW/Z235oP8Mfes57iYi8JiLdIvKjJBWUUdiVlbuAeSJSad9HmYg8JCLt9rkeEpG6hOf0T/bz6ReRxz12IyIfEZE3RaRTRP7Oez0RyRGRH4hIq/33AxHJsbetF5EWEfkrETlqe77XiMh7RGSfiHSJyN9OdE/JEJFTbLt77PfrKs+2n4vIT0Rkg4gMAhfadn5fRA6KyBER+S8RybP3r7CfSY9t07MiomXrFKAPcRZgjBkG7gU+6ln9fuBVY8xOe3nQ3l6KJTL/R0SuSeH0n8YSupVYtfzrE7YftbcXA58A/k1EVhljBoHLgVZP7bbVe6CInAjcDXwRqAQ2AA+KSHbCfVwGLAROAz4+hp2XAL81xsRSuKexOBtoAuYCtwC/BW5KsOUZY8xREVkF3AZ8BigHfgo84BS+XowxLcCbvFWBOB94FnguYd2mJMf+nb3v5+1n+HnP5iuBM4HTbdvePdEN2s/2o0An0G2vzgJuB+YDDcAw8MOEQz+I9f3OBbKBr9jnWwr8BMtbr7WfRZ3nuL8DzgFW2HaeBXzNs70ayAXmAV8HfgZ8GDgD69l8XUQWTXRfCfcYAB4EHrft/QJwl4iclHA/3wKKgD8C/wKcaNt5gscegC8DLVjvaBXwt4DmkJsCVKBmD78AbnBqfViFkNuPZIx52hiz2xgTM8bswhKGC5KcJ5H3Az+wvZEu4J+9G40xDxtjXjcWz2AVCuuSnSgJNwIPG2OeMMaEge9jNdGd59nnP4wxrfa1H8QqQJJRDhxO8bpj0WqM+U9jTMQW/V8RL1AftNeBJdw/NcZsMcZE7T67IFZhnIxngPPtmvdZwPNYwuOsW2PvMxm+Y4zpMcYcxGq2G+vZALxfRHqwxOfTwPVO068xptMY8xtjzJAxph+r4E58N243xuzzVIaca10PPGSM2WSMCQJ/D3grCR8CbjHGHDXGtAP/iCVmDmHgW/b3fw9WE+S/G2P6jTF7gD1YFZOx+Irt2fSISIe97hyg0H4+IWPMRuAh4r/L3xtj/mRXaIL2M/mSMabLfgbfBj7gsbEGmG+MCRtjnjWa5HRKUIGaJRhj/gi0A1fbNc4zeaswRUTOFpGn7GacXuCzWIXBRNQCzZ7lN70bReRyEXnebvroAd6T4nmdc7vnswuLZqzaq0Ob5/MQVsGTjE6sQuR4aE5Y3gjk2c9uPlahfL+9bT7wZU/h2APUY91TMjZheUmnAk3GmCGsmruzLg/YMkl7U302APcaY0qxPICXsTwUAEQkX6xAjjdFpM+2tVTi+2bGulbc+2F7zp2efeO+Y/uz9xl1OsEgWOIJcMSzfXiC+/q+MabU/nPeu1qgOcGbfpP498r7XVcC+cALnu/yUXs9wPeA/cDjYjVjf3Uce5RJoAI1u/glluf0EeBxY4z3h/4r4AGg3hhTAvwXMGGfBZZXUu9ZbnA+2M1Zv8HyfKrsAnCD57wT1TJbsQp653xiX+tQCnYl8iTwvnH6Bgbt//meddUJ+8TZaxdw92LVvD+I5Sn025ubsWr+pZ6/fGPM3WNcfxNWE9cVWJ4TWN5Bvb1umzFmZIxjp6y2bozpwGqW/IaIOIL+ZeAk4GxjTDFvBW9M+v0QkXwsb9Yh7jvGen/imnrTQCtQn/AuNBD/XnmfaQeWEC7zfJcldlAJtjf3ZWPMIuC9wP8VkYvTfA+zAhWo2cUvsfpiPo2nec+mCOgyxoyIyFlYBW4q3Av8uYjUiRV44a09ZgM5WJ5bREQuB97l2X4EKBeRknHOfYWIXGz3G3wZq7nluRRt8/KvWP1gv7C9HURknoj8q4icZjcvHQI+LFbgxydJLbrvV1hNkR/C45Fi9ZV81vauREQKxApEKUp2EmPMfqzn8RfYAmU3E22x143qf/JwBJhUP8x4GGNeBR4D/speVYRVQPeIFQ35D5M43X3AlSKy1u7fuoX4cudu4GtiBZhUYPXr3JnkPFPJFqwKyV+JFb6/HktY7km2s10R+RlW/6kTADNPRN5tf75SRE6wK1B9QNT+U44TFahZhB299hxQgOUtefn/gFtEpB+rkLg3xdP+DKsw2wnswAoccK7XD/y5fa5uLNF7wLP9VawCqsluOolr/jLG7MXqEP9PrFrse4H3GmNCKdrmPVcXVt9VGNhi3+cfgF6s5hmwhPsvsZqglpGCEBpjnMKuFisq0Vm/3T7fD+1738/YARwOm7Cajf7kWfcsVkf+eAL178D1YkXY/cdENqfI94Cb7QL5B1hNjB1YfWOPpnoSu5/oc1jifRjrWXjHvn0T2A7sAnZjvUPfnAL7x7MpBFyFFaTTAfwY+Kj9Po7FX2N9h8/bzZxPYnmVAEvs5QGsoQw/NsY8nR7rZxeifXmKoijKTEQ9KEVRFGVGogKlKIqizEhUoBRFUZQZiQqUoiiKMiOZyoSXGaWiosIsWLAg02YoiqIoE/DCCy90GGMqJ9rvHSNQCxYsYPv27Zk2Q1EURZkAEXlz4r20iU9RFEWZoahAKYqiKDMSFShFURRlRqICpSiKosxIVKAURVGUGUnaBEpEbrOnaX55jO0iIv8hIvvFmrJ7lWfbx8Saqvo1EflYumxUFEVRZi7p9KB+jjUV91hcjpUFeAlwM9a00HjS+Z+NNbPoP9jTOCiKoiiziLSNgzLGbBKRBePscjXwS3vOm+dFpNSeIG098IQ9PQIi8gSW0I010du0EgqFaG1tJRwOAxAOh+nt7aW3t5dgMDjusTk5OZSWllJUVMTw8DA9PT0MDg6Oe4yiKMpM48wzz6S2dqzJoaeOTA7UnUf8tMot9rqx1o9CRG7G8r5oaGhItsuUYIzhr//6r7nzzjtpa2tDpyhRFGU2861vfYu//du/Tft1MilQyaaLNuOsH73SmFuBWwFWr16dNtX4yU9+wve+9z1OPPFETjzxREpKSvD7rUcnIuTm5pKbm0txcTHWpJrJCYVCdHd3EwwGCQQC5OTkUFRURCAQSJfpiqIoU8769eun5TqZFKgWoN6zXAe02uvXJ6x/etqsSmDz5s38xV/8BUuWLOH73/8+733vezNliqIoyqwik2HmDwAftaP5zgF6jTGHsaYPf5eIlNnBEe+y10077e3tXH311RQWFnLTTTdxxRVXZMIMRVGUWUnaPCgRuRvLE6oQkRasyLwAgDHmv4ANwHuA/cAQ8Al7W5eI/BOwzT7VLU7AxHTz4x//mI6ODm6++WY++clPkpWlw8YURVGmi3RG8d00wXYDfG6MbbcBt6XDrsnw+OOPU11dzYUXXsj8+fMzbY6iKMqsQl2CMRgZGWH79u0sWLCA888/P9PmKIqizDpUoMZgy5YthEIhFixYQFVVVabNURRFmXWoQI3Bxo0bERFWrlypfU+KoigZQEveMXjssceorq7mzDPPzLQpiqIosxIVqCSMjIywY8cOFixYoMERiqIoGUIFKgnPP/884XCYBQsWMG9e0ixLiqIoSppRgUrCE088gYhw6qmnkp2dnWlzFEVRZiUqUEl44oknqK6u5vTTT8+0KYqiKLMWFagERkZG2LlzJwsWLGDhwoWZNkdRFGXWogKVQEtLC6FQiKqqKurr6yc+QFEURUkLKlAJdHVZaf/y8/MpLCzMsDWKoiizFxWoBByBysvLy7AliqIosxsVqAQcgZo7d26GLVEURZndqEAl4AhUXV1dhi1RFEWZ3ahAJeAIlAZIKIqiZBYVqAS6urrIycmhsrIy06YoiqLMalSgEujs7CQvL4+ysrJMm6IoijKrUYFKoKOjg7y8PI3iU5QZRCQSob+/f9qva038rWSKtE35/nbF8aBUoBQl8xhj6O3tpaurC2MMfr9/Wn+bhw4dorS0VMdEZggVqAS6urpUoFLAGEMoFGJkZISCggL8/ql7lWKxGMYYfD7flJ1TefsRDoc5fPgwkUjE9WQ6Ojqoq6tDRCY83hhDd3c3+fn55ObmuutjsRhDQ0OEQiGCwSC5ubkUFBSMSgztbO/o6KCgoCClaybinCMcDgNQVlZ2TOcZj2g0yuDgICMjI5SWlr6jElyrQCXQ3d3NokWL4l5oJZ7h4WEOHz4MWIXAwMAAtbW1U/bD6+joYGRkhPr6+in/MSszF2OM+30771hiE1s4HGZ4eJj8/PwJz9fV1UVvby89PT1UV1eTn59PMBjk8OHDbiUIYGhoiO7ubgKBQJz49fX1AZag9fX1UVJSkvK9hMNhOjo6GB4ejrs3v99PcXGxu667u5uCggJycnJSPreDMYYjR44wODiIiLi/xcLCQsrLy98RFby0CpSIXAb8O+AD/tsY852E7fOB24BKoAv4sDGmxd4WBXbbux40xlyVTlvhreaEvLy8KfUI3s5EIhF8Pl+cUPT09MQVHMFgkIGBAYqKio77euFwmIGBAQAGBwdnTNNKOBymvb2dioqKpDVUx6MMBAJkZb3VtRuLxQDi1qUDYwzDw8OEQiHC4TA+n4/CwsIpq007fUBFRUWjfhvO76a7uxsAESErK4vs7GxycnIoLi4es7CMRCIMDAwwMDBAMBhERPD5fESj0aT9P8YYOjo6qK+vZ3BwkI6ODvLz86msrIx7R/v6+ujt7XXP0dbWRlFREf39/WOeNxwO09vbS2lpKcYYt8/LGENXVxdFRUWICL29vQCUlpa6xweDQVpbW9379wqg9xqdnZ0UFBTg8/no6+uju7ub3t5eGhoa8Pl8GGM4evQow8PDlJSUJH3eDp2dnQwNDbnndv739/czODhIdXW12xLk3J/zfEUkaeXPsdERZ+e79Pl8ZGdnU1JSckxieqykrRQWER/wI+BSoAXYJiIPGGMaPbt9H/ilMeYXInIR8M/AR+xtw8aYFemyLxn9/f1Eo9F3bPNeLBabVEFpjKGlpYXS0lL3x+gUhIn7OQXFZGtt0WiUrKws98fS0dER15xzrE0rYzEyMkJnZyeRSITS0lKKioomfCbRaJRDhw4RjUZpaWmhqqqKgoICd3ssFuPIkSMMDw+7/SQ+n49wOOwKFIDf76e8vHxS99Tb24vP53OPcWrzWVlZbk0crHfX++zAqkj4/X63hh4IBIhEIoTDYbKysigsLEzpfRgeHqatrY1YLEZ3dzdlZWUUFxcTjUaJRCJ0dHTENcM5zywcDjM4OEhvby/z5s0jEAi42719S86y8z8SiYxrTyQSoaWlhXA47HoNwWCQ6upqIpEIw8PDoypRxhi30B0Lx6MpLi4mGAyOOt7x7B37srOzyc/PJxaLuV7ZRDhiV1xcTGdnJ2C9P62trcybNy/uPeru7qarq4v8/HzKysrIycmJ8+76+vrGDOJwbHLeEa8wO95cTk4OeXl5FBUVud9NV1dX3HmNMcRiMSKRiFsRzc7OZu7cudPSlJhON+EsYL8xpglARO4Brga8ArUU+JL9+Sngd2m0Z0LeyXn4hoaGaGtro6GhIWXvsK+vj2g0Sm9vLyUlJYjIKHFyiMVitLe3U1VVlXLhG4vFaG5uJisri5qaGmKxWNz5Y7GY65k5BZfP5yMrK4tIJOLWkkXE7UcoLCx0r2+Moaenh2AwCFgFWygUcn98nZ2ddHZ2kpub69b2vcc75zh8+DDRaNRdPnLkCAUFBeTm5uLz+ejo6HC3O9dJVshGIhGOHj3q1kSdH38gEHDP5cURU7A8sKKiorjCw+/3k5+f74pEshp7OBymp6dn1D2JCB0dHW7lI1GootEowWCQ4eHhOE/EW3B6n/N4OMLuNAOHw2G6u7vjvovJ4HiricsHDx50m7qOFUc4kwnUwMBA3Lq2tjbq6+vp7OxMSZyc8/T39486Vzgc5s0338QYE/eswfrtDg8Pk5WVRSAQwO/3Mzg4OOF9OqI8lsc4MjLCyMgIPT095Ofn4/f7xxU957hgMEhPT8+0pINLp0DNA5o9yy3A2Qn77ASuw2oGfB9QJCLlxphOIFdEtgMR4DvGmFHiJSI3AzcDNDQ0HLfB71SBikajHDlyxP1xeMd4DQwMMDIyEtc+7vzInSabaDRKKBQiJydnzCYSsH5Izc3NVFZWjnqG4XCYvr4+ysrK3MKwp6eHWCxGNBqlubkZv98/qlDo7Ox0RdJp9kksbMFqDnR+yE5zj9MHMZa9zvrh4WGGh4cRETo7O137nZq4tzB0jhsYGHDb/lMtnJxjg8Eg7e3t7rLzvAOBADU1NQQCAVcIHRuj0Sg9PT1x53IKyPb29pQKq2TLPT099PX1UVNTQ05OjtuU6TyPsQq3ZOccj1gsRktLS8qidqwc73m97/1E5zbGcOjQoaTNeRNdI9m5JhKGaDQaVxFK9Vqp7OPtx5pJpFOgklWjE+/+K8APReTjwCbgEJYgATQYY1pFZBGwUUR2G2NejzuZMbcCtwKsXr36uJ/sO1GgnILOKUSdNnYRcWv0zkspIgwNDVFdXU1fX597jFOrrKysZHBw0D13f38/b7zxBqeeeqr7cjuRV46XUFBQQF9fnxsmHAqFqK6udgtcb2HnRDp5icVi7rHeexrrXp3+q+zs7HHFaazjvWI+0Q92okJlomslfg6Hw66nMTQ0NGFh5DTBHq/H4DRhFhQUxNXM01FYZboA7O7u5tFHH2X58uUsX778uJuPJysYM5lMfzfJSKdAtQDehHZ1QKt3B2NMK3AtgIgUAtcZY3o92zDGNInI08BKIE6gphpHoMrLy9N5mUkxODhITk7OpII2YrEYIyMjRKNR141P3JaXl0d3d/eogtKJnkpsfnGig7z7PvLIIzQ3NxMKhVi9enXcNsdLOHr0aFxBPzw87PYBpVq7mwxekTpe8ZjM8a+99ho5OTnH7cnHYjEOHTqU8vW93tvQ0BCHDh3i6NGjnHTSSVRUVKR8Xe9zS8bg4CA+ny9t0a39/f1u85Jji9PXtmLFirj+K4doNEpfX59b4ZqIYDDI7373O7q6umhtbeXgwYNceumlo/pS0lFQDw4O4vf7JwwwCIfDvPTSS5xyyikzJkAok6RToLYBS0RkIZZn9AHgg94dRKQC6DLGxIC/wYroQ0TKgCFjTNDeZw3w3TTaCrwlULW1tem+VEoMDAy4/R3V1dUpH9fe3u56OsmaEnp7ewkEAjQ3N9PY2MiyZcviQl+9gpZ4Xud8+/bto7m5maKiIp599lnmzZtHTU3NqGsl2uC0i4/FgQMH2LRpE8uWLWPVqlXHVMOd6gLGGMOePXvIz89n0aJFo7a/8sorbNiwgaysLK699lrmz5+f0nmdAjYxrZZjf0dHB01NTZx++unjFmyxWIyHH36Yffv2ueu2bt3K6tWrOeecc5IW7qnS3d3Nli1baGxsxOfzcfrpp7N69Wqys7Pp6+sjFApRVFQ0qu8OrCbI1tZWVq5cOeb32NXVxdatW2lsbBzze3vxxRe56KKLOOGEE9x10WiU3/3udxw4cICKigqWLl3KsmXL4sLPQ6EQvb29lJeXIyI88sgjdHd3c/3119Pa2srmzZtpa2vj4osvZsGCBaOuu3PnTnbv3s3atWuTbk/EGOMK35IlS9wKwoEDB3jwwQfJysri/PPPH9Nzi0ajPPTQQzQ1NbFr1y5uuOEG93fZ19dHIBCIa905evQor776KoWFhRQVFVFTUxMnasFgkK6uLgoLCykoKEh7JGk6kHS6dSLyHuAHWGHmtxljviUitwDbjTEPiMj1WJF7BquJ73O2KJ0H/BSIYaVj+oEx5n/Gu9bq1avN9u3bj8veb3/72/zd3/0d//u//8v1119/XOc6XrzjQESEurq6pFEzg4ODZGdnu4XQyMgIra2t4xbSIkJ+fj533nknTU1N+Hw+li9fztlnn51SqHg4HOb2228nNzeX66+/nrvuugsR4SMf+cgxh6BGo1Gee+45tm7dSk5ODsFgkEWLFvHud787pTEvY9npPMPE51FbW5tSFNLQ0BCPPfYYTU1NiAiXXXYZS5cudbcfOHCA+++/n9raWkZGRujt7eXGG28kEAiwdetW2traWLx4MUuXLo3zzKPRKPfffz9vvvkmq1evZu3atXFBEm1tbdx3330Eg0EKCgpYv349ZWVlNDY20tzczMqVKzn11FMBeOqpp9ixYwdnnHEGS5YsoaSkhD/+8Y/s2bOHoqIizj77bJYtW5ayF97S0sKbb75Jc3Mzra2t+Hw+Tj31VILBIK+88krSd8vn81FfX88555xDbW0t27dv549//COxWIxzzz2X8847L27/zs5Onn/+efbu3YvP5+O0005j4cKFgBUQUlBQQHFxMUePHuXJJ5+ko6ODRYsWcfHFF1NUVMQjjzzCK6+8wooVKzhy5AiHDx/G7/dz2mmncdppp7F//362b9/OyMgIgUCA0tJS2tvbufDCC1m1apV7n48//jjd3d2cdNJJrF+/3i3gDxw4wG9/+1t8Ph+RSISTTjqJ1atXU1xcTF5eHsFg0A3S6e/vp7e3l6amJrefMCsrizPOOIOioiKeeuopd3jCoUOHqKmpYeHChRQXF1NWVsbcuXPx+Xw89thj7NmzhzPOOIPdu3eTl5fHpZdeyssvv8zevXspKCjg2muvpbKykkOHDvHb3/42rn9URJg/fz6LFy+mpaWF/fv3u02QWVlZlJeXU19fT01NDaFQiP7+fnw+HytXrnR/t319fWzfvp3u7u64CkhxcTHFxcWUlJRQXFzM4sWL3e/rWBCRF4wxqyfcbya2Ox4LUyFQX/nKV/iP//gPNm7cyNq1a6fIssmTTGTy8/NHeSiRSMSNXHLCeA8ePDgqgiwajbJhwwZyc3O55JJLyMrK4siRI9xxxx2sWLECYwy7d++msLCQD33oQxMKwnPPPcfmzZu58cYbqauro7W1lXvuuYeGhgauuuqqlMNPGxsbeeqpp9zgh0gkwqmnnsqFF17I7t272bRpE9nZ2SxdupSlS5cSiURobGzkwIEDNDQ0cPbZZ485eHL//v1s3LhxzPxtfr+fE044gcWLF+P3+92w25KSEvLz82lvb6e5uZkdO3YwPDzMunXraGpqcpuFqqqqaG5u5rnnnqO0tJQbb7yRUCjE3Xff7WYO8Pl8VFdXc+jQIYwxNDQ0cNFFFzFnzhw2bNjAq6++yvz583nzzTepqalh3bp1lJSU0N/fz29/+1vy8vJYv349mzdv5ujRo4AlBMXFxXR3d3PmmWdSVlbG448/zqpVq7jwwgvj7rG5uZlnn32Ww4cPU1BQwPnnnx8nrlu2bOHVV1/l4osvpq6ujlgsxjPPPMOOHTsQEaqqqpg/fz4rV650w+p7enrYs2ePG1DjBM50d3fT2NjI8PAwxcXF9PX1sWTJEnw+H6+++ipXXXUVS5YsoaOjwxWmQCDAihUrOOOMM+LC9hOJRqPs2LGD5557DrDmajtw4ABr1qzhnHPOASzB27ZtW5wntmjRIpYsWUJbWxuHDh2ivr6eCy+8MM57iUQibN26la1bt+Lz+VizZg3z58/n7rvvpqioiPe///289NJLbNmyJa6wTwyM8fv91NTUsGzZMurq6ti8eTN79uwBYOHChVx55ZUEAgH27NnD5s2b41oR/H4/paWldHR0cN5553Huuedy+PBhfvOb3xAMBgkEAixfvpzXXnuNUCjEOeecw+bNmyksLOSGG27A5/PR29vL66+/TmNjI/39/eTm5nLyySfT0NDA0NAQvb29HDlyhNbWVrd8cJrenQpQf3+/+4zLy8spLi4mOzvbbX719ku///3vZ/369WN+ZxOhAnUMfPzjH+f+++/n2Wef5bTTTpsiy1InFovR2dmZNFLOESGvh3LkyBG33yBZGLLDxo0befHFFwG44oorOPnkk3nooYd44403+PSnP01ubi6HDx/m17/+NbW1tVx33XUYY3j22Wc5cOAA119/vetZdXV1cccdd7B48WKuvPJK9xq7d+/miSeeoKKigve9730TemI9PT384he/oLy83J0csq6uLq4Z5+jRo2zevJmmpib3h+H3+5k3b54bHLBo0aJRgtjf309zczPl5eWsWbPGbRZxBieGQiH279/P3r17x2zOdJg7dy7vete7qKqqIhwO8/vf/54333zT3V5ZWcm1117r1ry7urrYsGEDDQ0NrF69mvz8fAallKBbAAAgAElEQVQHB2lsbGTLli2Ew2Fqa2tpaWlh3bp1nHXWWezdu5fHH388rjZcWlrqNvHEYjFeeeUVotEoS5YsIScnh40bN7Jz507AimC97rrrkjbhGGM4ePAgf/rTnzh8+DBnn302a9as4bnnnuP55593x0ade+65tLW10dTUxKpVqzjvvPMm7Q2Hw2F27drFyy+/zOmnn87pp59ONBrl3nvvpb29nfnz5/P6668TCARYuXIlZ5xxxqS8476+Pp566in279/PihUruOiii0Y1lfX09LBv3z7mz59PVVVVyufu7u5m48aNHDhwwB1k/OEPf9itAPX399PW1kZ/fz/9/f3k5+dTUlLiDqbNy8sbZUtLSwtHjx5lxYoVo74bZ+BzR0cHLS0ttLa2smDBAs477zz3PO3t7bzxxhssX76c/Px8+vv7uf/++2lvb6esrIz3v//9o/qpjLHGWZWWliYdkxiNRt2hFYWFhRw5coQnn3zSrQAtXryYiy66KG6MnUMsFmNwcJC+vj4aGhpSavYcCxWoY+DKK69k+/btPPvssyxZsmSKLEsNZ/DheCGrubm57liSUCiUUgTX7t273Rr24cOH6e7u5uqrr+bee+9l9erVnH/++e6+jY2NPPLII5xyyil0dnZy9OhRfD4fNTU13HDDDcRiMX71q1/R39/PRz/60VEi9MYbb/Dggw+Sk5NDXV2dGzpdW1tLXV2d25kdi8W499576ejo4GMf+9iEYjY8PMxrr72Gz+fjhBNOcGvt27Zto6mpaVToeVZWFqeeeiqrVq0ad+BwJBJxIwSdAJG+vj4GBweZM2cO9fX1o2r2kUiEXbt2kZeXR11d3aSyZwwNDbFp0ya3GeeCCy5w7R4aGuLIkSP09fUxMjLCsmXLxu0kN8bw4osv0tTUxBVXXDFh5Gk0GuUPf/gDu3fvpqKigo6ODpYvX84FF1zAk08+yd69exERLrzwQlauXJnyPaXCwMAAd911F+FwmJUrV7Jq1arjipTt6elxx+VNJcYYXnvtNV544QXWrl07IyctDYVC7Nq1i1NOOWVcr3MyxGIx9uzZQ0FBQdI+1mQUFRUd1zgoFahjwKlFPvvss9M65bszRiRZmLUXZ6xSVVUVnZ2dYw6aBavAe/XVV9m0aRN1dXVce+219PT0cMcdd7gF+p/92Z+5haDzHjzzzDNs376d3Nxc3v3udxMMBnn00Uc577zzGBoa4qWXXuJ973vfmC+y02fgeCbDw8Pu59LSUpYuXUo0GmXLli1cdtllLFu2bNLP6+3O4OAg+fn5055n0BjDtm3b3BaCSy65xG3m2bdvH/n5+WkrlJ00RjMlkelMHPPzdmK6BEoTznlwMplPZ6JYY4wb1g2MKrQ6OztpbGzklVdeoaioiMsuu2xUH1MoFKK1tdXNodXV1cXBgweJxWLU1NRwxRVXkJWVxZw5c1i7di1PP/00K1ascCN7AoGAO3J+3bp1zJkzh/nz57tu/sGDB9226TPOOCNOnBJ/6HPnzuWDH3wrWNNpcmhubmbfvn3ueZzAgdnIVNV8x2KswbAiwllnncXy5cvjmqREhJNOOimtNk1n/raJcJp6VaBmPipQHrq7u6mvr5/SgbpOJIyTC82ZosLxlpxU+ffddx8+n49rrrmGnJwctw9o27ZtbnTO4cOHufPOO7nooovIzc2lpaWFlpaWuIwDgUCA4uJiVq1axdKlS0dNXb9q1SoKCgpYuHAhIuLmVTt8+DAjIyNu85iXiy++mCNHjpCdnc26detG3aOT3DMZIkJ5eTnl5eWsWLGCvr4+3njjDU488cR3VKZyEaGwsHBUCptUjx3rmMkUpE50ZklJidsxnuzYyfT7ONd38gs6npB3cPdUFvROctJUszNM9voiQklJCbm5uXG/m5lEurNtvJ1QgfLQ19dHbm7ulHlQTsr9xD4S74sXiUT4/e9/T1tbGyLCfffdx3XXXcfmzZvZsWMHp556KmvWrHEzMjz88MM8+uijgCUMVVVVnHXWWdTV1VFVVUVubu64Bb+IcPLJJ7vLTobmmpoaWlpaRqX1AdwO46ysrFGdvXl5eVRUVLj9ZxNRXFzM6aefPq59iYXfTKntjmWHiDBnzhyKi4tTypHmpbCwEGMMQ0NDcfftJIMtLCx0m3PHO29WVha1tbWup5KXl0cgEEiaoy+V+3RCvQsLC8nJyXG/d6dz36l0ZWVlTSoX3VjXS0xA6p2uYjz7s7OzmTdvHpA8YW6yaznptnJzc8dtJk8Fn8+XVEzHe1f8fv+YA9WdCl1ippVjrQD5/X6ysrLcxLpvN1SgbIaHhwkGg+Tl5U3ZPCreFD3JXo5YLMYjjzzCwYMHufzyy8nOzubBBx/ktttuY3h4mFWrVrF+/XpXcIqLi7nxxhvZt2+fO3j3eAZhlpSUuAWPiFBdXU1zc3NSW5ONoXF+NE4ghBNOPVmcZK9z5851k1dGo1HmzJlDQUGB691594eJa5h+v5/s7Oy4wjUxCeh4NjnXyM7OprCw0B1Ym3i8k9bJKfwSUzONV1iVlZURCATcyoGzzpsdwUk91dXVRSAQID8/n2g06obQOxGeif07xcXF+P3+uFRX492nQyAQYN68eUmjAv1+/6iBxbm5ubS2tib1ovPz88cUGaepzRF3b8XKyUvY3d09KuOJ9/i5c+fG/T6CweCY+SJFhMrKSve+5s6dy8GDByfloSZ+r9XV1YRCobiKaHZ2dtKKntN/nJubS0dHx6h3SUTc8V9gCW4kEol7J7ytL8lsSrTXye3Y399PZ2dn3JjAsSrNMwkVKJupzsPnTDUwHs899xz79u3jggsucPtjrrrqKh566CHOPPNM1q1bF/cSFRQUEAqFOOWUU475hfJ6JYljiAKBAHPmzBlVwI6FM4YCrD6GqqqqSTebOAWUU8AXFhaOil5zvDtnmghnUKzzg0skOzub2trapBUNZ3T90NBQ0gIHrHfAmaIiOzs7rhB3mmSdZxgIBKiuro4rJJ0C1YmAdHIeJoqEk0XduceOjg7mzJkzSmic7yrx+yovL2dgYMD1lpLhBD4cPnw4aS06KyuLvLw89558Ph+1tbWTyjqQnZ1NfX09hw4dGlV4zp071523yVswFhUVUVRUFDeFRDJKS0vdRMHJnknis6qoqCAYDMal6vJ+N97+P7/fT2Vl5ahku4meuzNGrri42N3XuQenxcXJGl9UVERFRQU9PT1xwuqImePhzpkzh4GBAfe+nGt4BbeqqopDhw6RnZ3tVljmzp3rjpF0hCsYDMZ54M75ysrK3OdTXFxMUVERg4ODrl1lZWUUFha6054kPi+n4miMcaeOcbJZTMXcb6mgAmUz1QLV2dlJa2srzz//PIsXL+akk06K6yg+cOAAW7ZsYfny5XF57BYvXsznP//5UYWrt7Z99OjRUS/kRPh8Purq6ohEIu48Qcm8opKSEnd+nYnIzc2NK8gKCgpGCdxENTynZjgejih1dnZSXl7uekZOJnbv+car/YMlpN6pPZxKhJONIy8vb8xjvQVEMBh0BzMmhrg7wlFVVYXP53Nz9HV2drpNNI4wO/j9/kmls3KulWy8SiJ+v5+6ujqam5tHCUhJSQllZWXuYNuamppjakFw3i+vSFVUVODz+dyJAp2s+YnzaY2HiFBRURGX1Bhwg36S7V9bW+tO3ui0iow171dRURE+n4+2tra4Qr+oqIihoSF30kwnlVN2djatra1uU5zDnDlzKCoqcgXBK6wi4g5q9to/d+5c2tra3P0Tp4PPzc0d1WzvjF8aGhpyM9CDNbi/q6uLYDDoCkni72qsCmBpaSm5ubnurMK5ublu/s9M9xOrQNlMpUCFQiGGhobYsWMHTU1NNDU18dRTT3HiiSdy1llnkZubyyOPPEJ5eTkXXXQR8FZzRywWS1pAOAUdvFWz8oqI80NO1gbvFPCOKI1XEDo1PWfKhfHEJVktqrS01J0Vt7CwkPz8/KTeg9OhP5E4OTjNI97jnWwOzjPLy8uLa8IZD6ePZbIRdU6zieNtJMNJC+PF5/Mxd+5cioqK3PFl0xkt6rwfTkHstRVwPZrjISsri3nz5rk1fOd8jrAfPXqUysrKSYeaFxQU4Pf73RlhnXd4rMJzst9tfn4+8+bNo6Ojg4qKCvd3lux7zMnJob6+HmNM3HvmiJd3uaKigiNHjowpps5EhLm5uWOWO8nGwjmBT4li5uQQjcVibnmSKrm5uaMy1cwEVKBsHIFKtcAcC2Os6S0ikQhvvPEGy5YtY8WKFbz88svs2bOHxsZGCgsLCYVC3HDDDQQCAbc2nZWVlVRgEpvjvP1FzstYWlrqznLquPBOZ3dNTc2kCgUnbUswGKSjo4NQKORex9unNlY0WGVlZVz0YE1NTVzqJsfTmczkhmPZ6YzbmUy29+PleJJu5uXl0dDQkJE2//z8fLKzs92KzbHMgDwRjkglBgY5nu2x4PQdtba2Ulxc7CZ/nUpycnJSti/Vd80RSW9fbyKJ/XmpMNG9vx2Two6FCpSNI1CTbWZxasNz5sxBxJokLxwO09zcTDAYZMmSJVRXV1NdXc15553HCy+8wO7du7nkkkuoqKhwxae0tJRYLEZHR8eoaxhjRtWkHI/CSZLpvOilpaUMDg4SDAYpLCykoqLimF9Y7482FosRCoXo7u5maGiI7OzslH+oTlNFW1sbOTk5lJWVTdlA1ekUpqlisrXbqaSiooLWVmvWm+OtjI1FOu4vLy+PBQsWTLmgphOnIqkcO2+/X3eacARqMhkkYrEY/f39iAjBYJCSkhJ37Mn+/fvx+/1x8wPl5+ezbt26UWOJHHHJyspyc7d5ycvLS1oQ5+fnU1lZGddG7TRBRSKRKR0c6YTl1tTUTJjxIhlOGpVMt2nPdpxO/al+P6aDt5M4KVODCpRNZ2en2zyRKt5pkl9++WUCgYDbfPP666+zcOHCCcPAE+dpcQZZepvDxsrYDSTtJPf5fGn9MR9raLuK08ygqqrK7bxXlJmMCpSNk+ZoMh3FzniLnTt38uSTT+Lz+bjxxhsREQYGBjjhhBMmjGJLFB8nMs4pQCorK9OeGkeZXaS7AqMoU8U7pzftOHEEKtUoPmfq9Jdeeoknn3ySBQsWUFBQwAMPPMDOnTsRERYuXIgx1jxATsYGL1lZWaOaWRzRcsJlp2u8gaIoykxDBcpmsgI1NDTE3r17+cMf/sDixYu5+uqrueaaaxgZGeHll1+mrq7OHUAZCASYO3euGxThkDiGxqG0tJT58+dPaxiyoijKTEMFymbZsmXU19enLAo9PT1s2rSJyspK3vve97qpWS677DIANzu0V/CccRVOlNNY3pGIaBOMoiizHu2DsvnBD37AN7/5zZQ8qFgsxgsvvEBvby/XXHONKyZz5szh5JNPpra21h15nni+8vJyd1zR8eTRUxRFeaejApVAKgLV19fH5s2bqa6ududGys/PJz8/P24EPTDKI/NmIlAURVHGJq1NfCJymYjsFZH9IvLVJNvni8gfRGSXiDwtInWebR8Tkdfsv4+l004vqYwNeeaZZ+jr62PNmjVuc11xcTE+ny/OK3LSsiTiZHhQFEVRxiZtpaSI+IAfAZcDS4GbRCRxCtXvA780xpwG3AL8s33sHOAfgLOBs4B/EJHJ5wSZBD6fj0996lMTCsfAwAB/+tOfqK2tZf78+dj2up6XN/2PBjkoiqIcO+msxp8F7DfGNBljQsA9wNUJ+ywF/mB/fsqz/d3AE8aYLmNMN/AEcFkabQVSyyLx+OOPMzAw4HpPgNvfBLgpfJxkqIqiKMqxkU6Bmgc0e5Zb7HVedgLX2Z/fBxSJSHmKxyIiN4vIdhHZ3t7ePmWGj0VfXx9//OMfWbhwoZvCKCsrKy6nmTN/ivNZURRFOTbSKVDJ8qgkRgZ8BbhARF4ELgAOAZEUj8UYc6sxZrUxZrU3e3a6ePDBBxkZGXFz6QUCAerr6+P6nbKyssjOznZnYlUURVGOjXRG8bUA9Z7lOqDVu4MxphW4FkBECoHrjDG9ItICrE849uk02johnZ2dPP/88yxdutRN0FpTU5O0z8pJTaS5zhRFUY6ddHpQ24AlIrJQRLKBDwAPeHcQkQoRcWz4G+A2+/NjwLtEpMwOjniXvS5jPPzwwwCsWbOGrKwsqqurxwyoKC4uZu7cudNpnqIoyjuOlARKRNaKyCfsz5UisnCiY4wxEeDzWMLyCnCvMWaPiNwiIlfZu60H9orIPqAK+JZ9bBfwT1gitw24xV6XMV5//XUWLVpESUmJO5X3WPj9/rfdVAaKoigzjQmb+ETkH4DVwEnA7UAAuBNYM9GxxpgNwIaEdV/3fL4PuG+MY2/jLY8qoxhj6O7uZv78+e405oqiKEp6ScWDeh9wFTAIbr/RrEqx3d/fTyQSobi4mIqKikyboyiKMitIRaBCxoqbNgAiMusmJ+rs7ASsviXNAKEoijI9pFLa3isiPwVKReTTwJPAz9Jr1szCEajy8vIMW6IoijJ7mLAPyhjzfRG5FOjD6of6ujHmibRbNoPo6OgAVKAURVGmk3EFys6n95gx5hKsdEOzkvb2dnJzc3XqdUVRlGlk3CY+Y0wUGBKRkmmyZ0bS2dlJcXGxzt+kKIoyjaSSSWIE2C0iT2BH8gEYY/48bVbNMLq7uykpKdFZbhVFUaaRVATqYftvVmKMoaenh4aGhqRzOymKoijpIZUgiV/YqYpOtFftNcaE02vWzKG/v59wOOxOSKgoiqJMD6lkklgP/AI4gJVlvF5EPmaM2ZRe02YGTgRfcXGxelCKoijTSCol7v8D3mWM2QsgIicCdwNnpNOwmcKRI0cAS6A0O7miKMr0kcpA3YAjTgDGmH1Y+fhmBY4HVVZWpgKlKIoyjaTiQW0Xkf8B7rCXPwS8kD6TZhYdHR3k5OToGChFUZRpJhWB+j/A54A/x+qD2gT8OJ1GzSS6u7s1QEJRFCUDpCJQfuDfjTH/Cm52iVkz2VFvb68O0lUURckAqfRB/QHI8yznYSWMfcdjjHEFSiP4FEVRppdUBCrXGDPgLNifZ8WMfQMDA4RCIUpKSlSgFEVRpplUBGpQRFY5CyJyBjCcPpNmDu3t7QDaB6UoipIBUnELvgj8r4i02ss1wI3pM2nm4AiUelCKoijTTyqpjraJyMlYc0EJ8OpsSXXkTFRYWFioHpSiKMo0M2YTn4icKSLVALYgrQK+Cfw/EZmTyslF5DIR2Ssi+0Xkq0m2N4jIUyLyoojsEpH32OsXiMiwiLxk//3XMd3dcTIwMICIkJeXp1O9K4qiTDPjlbo/BUIAInI+8B3gl0AvcOtEJ7bD0X8EXA4sBW4SkaUJu30NuNcYsxL4APHjq143xqyw/z6b4v1MKf39/eTl5eH3+zWLhKIoyjQznkD5jDFd9ucbgVuNMb8xxvw9cEIK5z4L2G+MaTLGhIB7gKsT9jFAsf25BGhlBjE4OEheXp427ymKomSAcQVKRJw+qouBjZ5tqUQMzAOaPcst9jov3wA+LCItwAbgC55tC+2mv2dEZF0K15tyVKAURVEyx3gCdTfwjIj8Hius/FkAETkBq5lvIpK1iZmE5ZuAnxtj6oD3AHeISBZwGGiwm/7+L/ArESlOOBYRuVlEtovIdifibioZHh4mLy9Ps0goiqJkgDEFyhjzLeDLwM+BtcYY4znmC2Md56EFqPcs1zG6Ce9TwL329TYDuUCFMSZojOm0178AvM5bEyZ6bbzVGLPaGLO6srIyBZNSxxjD0NCQ2welKIqiTC/jlrzGmOeTrNuX4rm3AUtEZCFwCCsI4oMJ+xzEaj78uYicgiVQ7SJSCXQZY6IisghYAjSleN0pIRKJuB6UCpSiKMr0k7aS1xgTEZHPA48BPuA2Y8weEbkF2G6MeQDLQ/uZiHwJq/nv48YYY0cN3iIiESAKfNYTsDEt9Pf3Y4zRPihFUZQMkVbXwBizASv4wbvu657PjcCaJMf9BvhNOm2biN5eq5stPz9fx0ApiqJkgAlLXhH5vIiUTYcxM4n+/n4AHaSrKIqSIVIpeauBbSJyr50ZYlaMWO3r6wNUoBRFUTLFhCWvMeZrWEEK/wN8HHhNRL4tIovTbFtGcTyo3NxcFShFUZQMkFLJa4eYt9l/EaAMuE9EvptG2zLKwIA1BZZ6UIqiKJlhwiAJEflz4GNAB/DfwF8aY8L2gNrXgL9Kr4mZYXBwEJ/PR3Z2dqZNURRFmZWkEsVXAVxrjHnTu9IYExORK9NjVmbxDtLVEHNFUZTMkErb1QbAHYMkIkUicjaAMeaVdBmWSWKxmDtIV5v3FEVRMkMqpe9PgAHP8qC97h1LNBpVgVIURckwqZS+4snDhzEmRpoH+GYab5ojFShFUZTMkErp2yQify4iAfvvL5jmvHjTjdeD0j4oRVGUzJCKQH0WOA8r4WsLcDZwczqNyjThcJiRkREVKEVRlAwyYVOdMeYoVibyWcPQ0BCACpSiKEoGSWUcVC7WvE3LsKbDAMAY88k02pVRnCwSmihWURQlc6RS+t6BlY/v3cAzWBMP9qfTqEwzODgIaBYJRVGUTJJK6XuCMebvgUFjzC+AK4BT02tWZvGmOdImPkVRlMyQikCF7f89IrIcKAEWpM2iGYB6UIqiKJknlfFMt9rzQX0NeAAoBP4+rVZlGBUoRVGUzDOuQNkJYfuMMd3AJmDRtFiVYYaGhggEAvj9fhUoRVGUDDFu6Wtnjfj8NNkyYxgcHCQvLw9jjAqUoihKhkil9H1CRL4iIvUiMsf5S7tlGcTJIgEwSyYQVhRFmXGk0gfljHf6nGed4R3c3OdMtSEiKlCKoigZIpUp3xcm+UtJnETkMhHZKyL7ReSrSbY3iMhTIvKiiOwSkfd4tv2NfdxeEXn35G7r+NBEsYqiKJknlUwSH0223hjzywmO8wE/Ai7FyuG3TUQeMMY0enb7GnCvMeYnIrIUa+6pBfbnD2Blr6gFnhSRE40x0VRu6nhRgVIURck8qTTxnen5nAtcDOwAxhUo4CxgvzGmCUBE7gGuBrwCZYBi+3MJ0Gp/vhq4xxgTBN4Qkf32+TanYO9xEQqFCIVCKlCKoigZJpVksV/wLotICVb6o4mYBzR7lp1M6F6+ATwuIl8ACoBLPMc+n3DsvMQLiMjN2JnVGxoaUjBpYnQMlKIoyszgWErgIWBJCvsliy4wCcs3AT83xtQB7wHusMdepXIsxphbjTGrjTGrKysrUzBpYoaHhwHIzs7WNEeKoigZJJU+qAd5SxyygKXAvSmcuwWo9yzX8VYTnsOngMsAjDGb7czpFSkemxaCwSAAfr9fBUpRFCWDpNIH9X3P5wjwpjGmJYXjtgFLRGQh1mSHHwA+mLDPQaw+rZ+LyClYfVztWCmVfiUi/4oVJLEE2JrCNY+bkZERAAKBgAqUoihKBklFoA4Ch40xIwAikiciC4wxB8Y7yBgTEZHPA48BPuA2Y8weEbkF2G6MeQD4MvAzEfkSlpf2cWOMAfaIyL1YARUR4HPTFcHneFDZ2dnaB6UoipJBUhGo/8Wa8t0haq87M/nub2GM2YAVOu5d93XP50ZgzRjHfgv4Vgr2TSmOQAUCARUoRVGUDJJKCew3xoScBftzdvpMyizeJj4VKEVRlMyRSgncLiJXOQsicjXQkT6TMksoZGmx9kEpiqJkllSa+D4L3CUiP7SXW4Ck2SXeCWgTn6IoyswglYG6rwPniEghIMaY/vSblTlUoBRFUWYGE5bAIvJtESk1xgwYY/pFpExEvjkdxmWCUCiEz+dDRFSgFEVRMkgqJfDlxpgeZ8GeXfc94+z/tiYYDBIIBACdC0pRFCWTpCJQPhHJcRZEJA/IGWf/tzWhUIhAIKBzQSmKomSYVIIk7gT+ICK3Yw2m/SQTZzJ/2+IIlDbvKYqiZJZUgiS+KyK7sDKNC/BPxpjH0m5ZhlCBUhRFmRmk4kFhjHkUeBRARNaIyI+MMZ+b4LC3JSpQiqIoM4OUBEpEVmBNjXEj8Abw23QalUnC4TA5OTkqUIqiKBlmTIESkROxMpDfBHQCv8YaB3XhNNmWEUKhEIWFhSpQiqIoGWY8D+pV4FngvcaY/QB21vF3NOFwWJv4FEVRZgDjlcLXAW3AUyLyMxG5mOQz3b6jUIFSFEWZGYxZChtj7jfG3AicDDwNfAmoEpGfiMi7psm+aUcFSlEUZWYwYSlsjBk0xtxljLkSa+r1l4Cvpt2yDBCNRolEIipQiqIoM4BJlcLGmC5jzE+NMRely6BM4p1qQ7NIKIqiZBZ1EzwMDw8DKlCKoigzARUoDzrVhqIoysxBS2EPznTv2dnZ6kEpiqJkmLQKlIhcJiJ7RWS/iIwKrBCRfxORl+y/fSLS49kW9Wx7IJ12OjgelN/vVw9KURQlw6SU6uhYEBEf8CPgUqxp4reJyAPGmEZnH2PMlzz7fwFY6TnFsDFmRbrsS4YjUOpBKYqiZJ50uglnAfuNMU3GmBBwD3D1OPvfBNydRnsmRD0oRVGUmUM6S+F5QLNnucVeNwoRmQ8sBDZ6VueKyHYReV5ErkmfmW/hDZJQD0pRFCWzpK2Jj+RpkcwY+34AuM8YE/WsazDGtIrIImCjiOw2xrwedwGRm4GbARoaGo7bYBUoRVGUmUM6PagWoN6zXAe0jrHvB0ho3jPGtNr/m7BSLa1MPMgYc6sxZrUxZnVlZeVxG6xNfIqiKDOHdJbC24AlIrJQRLKxRGhUNJ6InASUAZs968pEJMf+XAGsARoTj51q1INSFEWZOaStic8YExGRzwOPAT7gNmPMHhG5BdhujHHE6ibgHmOMt/nvFOCnIhLDEtHveKP/0kUoFEJE8Pl8KlCKoigZJp19UBhjNgAbEtZ9PWH5G0mOew44NZ22JSMYDGoWCUVRlBmClsQeQqGQNu8piqLMEFSgPKhAKYqizBxUoDyoQCmKokrw2FAAABe9SURBVMwcVKA8qEApiqLMHFSgPOh074qiKDMHLYk9OAKlHpSiKErmUYHy4DTxqQelKIqSebQk9qBNfIqiKDMHLYltjDEqUIqiKDMILYltVKAURVFmFloS24TDYUAzmSuKoswUtCS2CYVCgGYyVxRFmSmoQNk4U21kZ2erB6UoijID0JLYRj0oRVGUmYUKlI0KlKIoysxCBcrGO5uuNvEpiqJkHi2JbRwPKjs7Wz0oRVGUGYAKlI3jQWmYuaIoysxAS2Ib7YNSFEWZWfgzbcBMwStQ6kEps51wOExLSwsjIyOZNkV5G5Obm0tdXR2BQOCYjleBslEPSlHeoqWlhaKiIhYsWKC/B+WYMMbQ2dlJS0sLCxcuPKZzpNVVEJHLRGSviOwXka8m2f5vIvKS/bdPRHo82z4mIq/Zfx9Lp53wVh+Uz+fTH6Qy6xkZGaG8vFx/C8oxIyKUl5cflxeeNg9KRHzAj4BLgRZgm4g8YIxpdPYxxnzJs/8XgJX25znAPwCrAQO8YB/bnS57vdO9649SUdDfgXLcHO87lE4P6ixgvzGmyRgTAu4Brh5n/5uAu+3P7waeMMZ02aL0BHBZGm0lGAy67aT6w1QURck86RSoeUCzZ7nFXjcKEZkPLAQ2TuZYEblZRLaLyPb29vbjMtbrQSmKklk6OztZsWIFK1asoLq6mnnz5rnLTn/xRHziE59g79694+7zox/9iLvuumsqTFbSQDqDJJKV9GaMfT8A3GeMiU7mWGPMrcCtAKtXrx7r3CmhAqUoM4fy8nJeeuklAL7xjW9QWFjIV77ylbh9jDEYY8aMur399tsnvM7nPve54zc2DUx0b7OFdApUC1DvWa4DWsfY9wOA901pAdYnHPv0FNo2CqeJTwVKUeL54he/6IrFVLFixQp+8IMfTPq4/fv3c80117B27Vq2bNnCQw89xD/+4z+yY8cOhoeHufHGG/n6178OwNq1a/nhD3/I8uXLqaio4LOf/SyPPPII+fn5/P73v2fu3Ll87Wtfo6Kigi9+8YusXbuWtWvXsnHjRnp7e7n99ts577zzGBwc5KMf/Sj79+9n6dKlvPbaa/z3f/83K1asiLPtL//yL3n44Yfx+/1cfvnl/Mu//AttbW185jOf4Y033kBEuPXWWzn77LP57ne/yy9/+UsAPvOZz/CFL3wh6b3t2rWLW265hWAwyJIlS7jtttsoKCg4/i/gbUI65XkbsEREFopINpYIPZC4k4icBJQBmz2rHwPeJSJlIlIGvMtelzYcD2q211gUZabT2NjIpz71KV588UXmzZvHd77zHbZv387OnTt54oknaGxsHHVMb28vF1xwATt37uTcc8/ltttuS3puYwxbt27le9/7HrfccgsA//mf/0l1dTU7d+7kq1/9Ki+++OKo444cOcKGDRvYs2cPu3bt4m/+5m8Ay0O79NJL2bVrFy+88AKnnHIKW7du5a677mLr1q1s3ryZH//4x/9/e/ceHVV1L3D8+yMJBGqEAMrCpl4eF6yBII/wKIEAtjeSSAUCiohFbMGiwgW7sODSupS2NFKKyGMh8lJZt1AqNwGpXgQJD5VnuBEwgIHIFYSCkEIIrzbhd/84J5PJkMQAGWbI/D5rZWVmn8fsvbMzv9n7nNmb3bt3X1W2iIgI0tLS+Pjjj9m1axft2rXjjTfeqK5qvCX4rQelqkUiMgYnsIQBi1T1CxGZDOxU1ZJgNRRYpqrqdWy+iPwWJ8gBTFbVfH/lFaBr166cPn3aelDG+Lieno4/tWzZks6dO3ueL126lIULF1JUVMSxY8fIyckhNja2zDF169YlOTkZgE6dOrF58+Zyz52amurZ5/DhwwB88sknTJw4EYD77ruPNm3aXHVcw4YNqVWrFqNGjeLBBx+kX79+AGzYsIFly5YBzjRqt99+O5s3b2bQoEHUq1cPgAEDBvDJJ5+QlJRUpmyfffYZOTk5dO/eHXA+RPfo0ePaK+wW5tcv6qrqB8AHPmkv+zx/pYJjFwHlf8zxg4SEBPLy8ixAGRPkvIe4cnNzeeONN9i+fTsNGjTg8ccfL/d7N7Vr1/Y8DgsLo6ioqNxz16lT56p9vD47VygiIoKdO3eydu1ali1bxty5c/noo4+Aq+8Krux83mVTVfr27cuSJUu+8/VrKhvP8mFDfMbcOgoKCoiKiuL222/n+PHjrFlT/VcCevTowfLlywHYs2dPuUOI586do6CggH79+vH66697hgH79OnDm2++CUBxcTEFBQUkJiaSnp7OxYsXKSwsZOXKlfTs2fOqc3bv3p2NGzeSl5cHwPnz58nNza328gUzm+rIhwUoY24dHTt2JDY2lrZt29KiRQsSEhKq/TXGjh3L8OHDadeuHR07dqRt27bUr1+/zD5nz54lNTWVy5cvc+XKFaZPnw7A7NmzGTVqFPPmzSM8PJx58+bRpUsXhg4d6hnKe/rpp4mLi+PgwYNlztmkSRMWLlzIkCFDPLfWT5kyhVatWlV7GYOVVKX7eiuIj4/XnTt3XvfxqkpeXh5RUVHceeed1ZgzY249+/bt49577w10NoJCUVERRUVFREZGkpubS1JSErm5uYSH2+f7qiivLYlIlqrGf9exVsM+rAdljPFWWFjIj3/8Y4qKilBVT2/I+J/Vsg+7ScIY461BgwZkZWUFOhshyboLPqwHZYwxwcHejX1YgDLGmOBg78Y+bIjPGGOCgwUoH9aDMsaY4GDvxj6sB2VMcPj73//Oo48+SsuWLYmNjSUlJYUvv/wy0NkqV7NmzTh16hSAZ2oiXyNGjOC9996r9Dxvv/02x46Vzqk9cuTIcr8YHCosQPmwAGVM4KkqAwcOpHfv3hw6dIicnBymTJnCiRMnyuxXXFxcwRkC57PPPrvuY30D1IIFC66aVzAYVDRVVHWz28x92BCfMWX99a9/5ejRo9V6zpiYGB5++OEKt2dmZhIREcHo0aM9aSXLW2zYsIFXX32Vpk2bkp2dTU5ODtOnT/fMUD5y5EjGjx/P+fPneeSRRzh69CjFxcX85je/YciQIUyaNIlVq1YRHh5OUlIS06ZNK/Pac+fO5auvvmLq1KmAEzSysrKYNWsWAwYM4MiRI1y6dIlx48bx1FNPXZX32267jcLCQlSVsWPHsn79epo3b15mDr7Jkyfz/vvvc/HiRbp37868efNYsWIFO3fuZNiwYdStW5ctW7aQnJzMtGnTiI+PZ+nSpUyZMgVV5cEHH+S1117zvN64ceNYvXo1devWZeXKlTRp0qRMnjZu3Mi4ceMA50P4pk2biIqKYurUqSxZsoRatWqRnJxMWloa2dnZjB49mgsXLtCyZUsWLVpEdHQ0vXv3pnv37nz66ac89NBDDB8+nNGjR/P1118DzqTC1T2ThwUoH9aDMibw9u7dS6dOnSrcvn37dvbu3Uvz5s3Jyspi8eLFbNu2DVWla9eu9OrVi7y8PO666y7+9re/Ac50RPn5+aSnp7N//35EhDNnzlx17sGDB/OjH/3IE6D+8pe/8OKLLwKwaNEiGjZsyMWLF+ncuTODBg2iUaNG5eYxPT2dAwcOsGfPHk6cOEFsbCw///nPARgzZoxn3aqf/exnrF69msGDBzN79mxPQPJ27NgxJk6cSFZWFtHR0SQlJZGRkcGAAQM4f/483bp14/e//z2//vWvmT9/Pi+99FKZ46dNm8acOXNISEigsLCQyMhIPvzwQzIyMti2bRv16tUjP99ZMGL48OHMmjWLXr168fLLL/Pqq696ZrQ/c+YMGzduBOCxxx7jueeeo0ePHnz99dc88MAD7Nu3r5K/6rWzAOXDelDGlFVZTydQunTpQvPmzQFnOYyBAwd6ZgJPTU1l8+bN9O3blwkTJjBx4kT69etHz549PVMWjRw5ssyyGN7uuOMOWrRowdatW2nVqhUHDhzw9AxmzpxJeno6AEeOHCE3N7fCALVp0yaGDh1KWFgYd911F/fff79nW2ZmJlOnTuXChQvk5+fTpk0bfvrTn1ZY3h07dtC7d2/uuOMOAIYNG8amTZsYMGAAtWvX9pSjU6dOrF279qrjExIS+NWvfsWwYcNITU0lJiaGdevW8eSTT3qW/WjYsCFnz57lzJkz9OrVC4AnnniizN9/yJAhnsfr1q0rc32soKCAc+fOERUVVWE5rpW9G/uwHpQxgdemTZtKZ2/wXZaiPK1btyYrK4u4uDheeOEFJk+eTHh4ONu3b2fQoEFkZGTQt29fiouLad++Pe3bt/f0aoYMGcLy5ctZsWIFAwcORETYsGED69atY8uWLXz++ed06NCh3KU9vJX3fnLp0iWeeeYZ3nvvPfbs2cOoUaO+8zyVzZnqvRJ4RUuJTJo0iQULFnDx4kW6devG/v37UdVrfr/zrvcrV66wZcsWsrOzyc7O5ptvvqnW4AQWoK5iPShjAu/+++/n8uXLzJ8/35O2Y8cOz/CSt8TERDIyMrhw4QLnz58nPT2dnj17cuzYMerVq8fjjz/OhAkT2LVrF4WFhZw9e5aUlBRmzJhBdnY2YWFhnjfZklV0U1NTycjIYOnSpZ5ew9mzZ4mOjqZevXrs37+frVu3VlqGxMREli1bRnFxMcePHyczMxPAE4waN25MYWFhmTv7oqKiOHfu3FXn6tq1Kxs3buTUqVMUFxezdOlSTy+nKg4dOkRcXBwTJ04kPj6e/fv3k5SUxKJFi7hw4QIA+fn51K9fn+joaM+CjkuWLKnwdZKSkpg9e7bneXZ2dpXzU1U2xOfDelDGBJ6IkJ6ezvjx40lLSyMyMpJmzZoxY8YMvvnmmzL7duzYkREjRtClSxfAuUmiQ4cOrFmzhueff55atWoRERHB3LlzOXfuHP379+fSpUuoKq+//nq5rx8dHU1sbCw5OTme8/bt25c333yTdu3acc8999CtW7dKyzBw4EDWr19PXFwcrVu39rzRN2jQgFGjRhEXF0ezZs3KrA48YsQIRo8e7blJokTTpk35wx/+QJ8+fVBVUlJS6N+/f5Xrc8aMGWRmZhIWFkZsbCzJycnUqVOH7Oxs4uPjqV27NikpKUyZMoV33nnHc5NEixYtWLx4cbnnnDlzJs8++yzt2rWjqKiIxMREz9pX1cWW23CpKidPnrzq7hdjQpEtt2Gqy40st2HjWS4RseBkjDFBxAKUMcaYoOTXACUifUXkgIgcFJFJFezziIjkiMgXIvJnr/RiEcl2f1b5M5/GmKvVlOF/Ezg32ob8dpOEiIQBc4D/AI4CO0RklarmeO3TCngBSFDVf4iI91rrF1W1vb/yZ4ypWGRkJKdPn6ZRo0Z245C5LqrK6dOniYyMvO5z+PMuvi7AQVXNAxCRZUB/wHvmw1HAHFX9B4CqnvRjfowxVRQTE8PRo0f59ttvA50VcwuLjIwkJibmuo/3Z4D6PnDE6/lRoKvPPq0BRORTIAx4RVX/x90WKSI7gSIgTVUzfF9ARJ4CngK4++67qzf3xoSwiIgIz0wNxgSKPwNUeeMCvgOS4UAroDcQA2wWkbaqega4W1WPiUgLYL2I7FHVQ2VOpvoW8BY4t5lXdwGMMcYEjj9vkjgK/MDreQxwrJx9Vqrqv1T1K+AATsBCVY+5v/OADUAHP+bVGGNMkPFngNoBtBKR5iJSG3gU8L0bLwPoAyAijXGG/PJEJFpE6nilJ1D22pUxxpgazm9DfKpaJCJjgDU415cWqeoXIjIZ2Kmqq9xtSSKSAxQDz6vqaRHpDswTkSs4QTTN++6/8mRlZZ0Skf+7gSw3Bk7dwPE1hdVDKauLUlYXDquHUjdSF/9WlZ1qzFRHN0pEdlZl6o2azuqhlNVFKasLh9VDqZtRFzaThDHGmKBkAcoYY0xQsgBV6q1AZyBIWD2UsrooZXXhsHoo5fe6sGtQxhhjgpL1oIwxxgQlC1DGGGOCUsgHqKosCVKTichhEdnjLmuy001rKCJrRSTX/R0d6Hz6g4gsEpGTIrLXK63csotjpttOdotIx8DlvHpVUA+viMg3XkvepHhte8GthwMi8kBgcu0fIvIDEckUkX3uEkDj3PSQaheV1MPNbReqGrI/OF8gPgS0AGoDnwOxgc7XTa6Dw0Bjn7SpwCT38STgtUDn009lTwQ6Anu/q+xACvAhzhyT3YBtgc6/n+vhFWBCOfvGuv8ndYDm7v9PWKDLUI110RTo6D6OAr50yxxS7aKSerip7SLUe1CeJUFU9Z9AyZIgoa4/8I77+B1gQADz4jequgnI90muqOz9gXfVsRVoICJNb05O/auCeqhIf2CZql5WZ/7Mgzj/RzWCqh5X1V3u43PAPpyVGUKqXVRSDxXxS7sI9QBV3pIglf0RaiIFPhKRLHf5EoAmqnocnIYK3Fnh0TVPRWUPxbYyxh22WuQ1zBsy9SAizXAmqd5GCLcLn3qAm9guQj1AVWVJkJouQVU7AsnAsyKSGOgMBalQaytzgZZAe+A48Cc3PSTqQURuA1YA41W1oLJdy0mrMfVRTj3c1HYR6gGqKkuC1GhauqzJSSAdp1t+omSYwv0dSisdV1T2kGorqnpCVYtV9Qown9LhmhpfDyISgfOm/F+q+t9ucsi1i/Lq4Wa3i1APUFVZEqTGEpHviUhUyWMgCdiLUwdPuLs9AawMTA4DoqKyrwKGu3dtdQPOlgz51EQ+11EG4rQLcOrhURGpIyLNcdZv236z8+cvIiLAQmCfqk732hRS7aKierjp7SLQd4sE+gfnLpwvce46eTHQ+bnJZW+Bc+fN58AXJeUHGgEfA7nu74aBzqufyr8UZ5jiXzifAH9RUdlxhjDmuO1kDxAf6Pz7uR6WuOXc7b75NPXa/0W3Hg4AyYHOfzXXRQ+coandQLb7kxJq7aKSerip7cKmOjLGGBOUQn2IzxhjTJCyAGWMMSYoWYAyxhgTlCxAGWOMCUoWoIwxxgQlC1AmZImIisifvJ5PEJFXquncb4vI4Oo4l3u++iLyrogccn/eFZH6Xtv/6M46/UevtCe9Zp3+p5TOWp9WXfkyxp8sQJlQdhlIFZHGgc6INxEJKyd5IZCnqi1VtSXwFbDAa/svcWaffr4kQVUXq2p7VW2P863+Pu7zMsvKiEh49ZfCmBtnAcqEsiLgLeA53w2+PSARKXR/9xaRjSKyXES+FJE0ERkmItvdHkpLr9P8REQ2u/v1c48Pc3s7O9wJN3/pdd5MEfkzzhchvfPy70An4LdeyZOBeBFpKSKrgO8B20RkSFUKLiK/E5F5IrIWWCwi4SIy3S3HbhEZ6bXvJK/0l920KBH5UEQ+F5G91dlbNKaEfXIyoW4OsFtEpl7DMfcB9+IsUZEHLFDVLuIs6jYWGO/u1wzohTO5ZqYbaIbjTIfTWUTqAJ+KyEfu/l2AtuosV+AtFshW1eKSBFUtFpFsoI2qPiQihW5P6Vp0ABJV9ZKIPAOcdMtRB9jq5qstcDfQFWfWhA9EpDvOvGuHVTUZnCHIa3xtY76TBSgT0lS1QETeBf4TuFjFw3aoO9+aiBwCSgLMHqCP137L1ZlUM1dE8oAf4sx32M6rx1EfZ96yfwLbywlO4ASG8qZ8qSi9qlaq6iX3cRJwr4g86pOvJJyZ7v/XTb8NaI2z9EKaez3rfVX99AbyYUy5LEAZAzOAXcBir7Qi3CFwd+LM2l7bLns9vuL1/Apl/6d8g4fiBJWxqrrGe4OI9AbOV5C/L4AOIlLLDXiISC2cnty+ygr2HbxfT4BnVPVjn3w9BPxOVRf6Hiwi8Tjzs/1RRFar6pQbyIsxV7FrUCbkqWo+sBxnktQSh3Gu+4CzWmjEdZz6YRGp5V6XaoEzieYa4Gl3KQNEpLU7k3xl+TuI04N5ySv5JWCXu606rAGeKblhQkTuEZG6bvovSvIoIjEi0lhEvg8UquoSYDrOkvHGVCvrQRnj+BMwxuv5fGCliGzHmb26ot5NZQ4AG4EmwGj3Ws8CnGtTu9ye2beULh9emV8As0TkIE5vZwtlA+qNmodzrSnbyRYngf6q+oGI/BDnmhTAOeAxnOtiaSJyBWd4cnQ15sUYAJvN3BhjTHCyIT5jjDFByQKUMcaYoGQByhhjTFCyAGWMMSYoWYAyxhgTlCxAGWOMCUoWoIwxxgSl/wc92JImo+f+gwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.model_selection import validation_curve\n",
    "\n",
    "# load data\n",
    "digits = load_digits()\n",
    "\n",
    "# create feature matrix and target vector\n",
    "features, target = digits.data, digits.target\n",
    "\n",
    "# create range of values for parameter\n",
    "param_range = np.arange(1, 250, 2)\n",
    "\n",
    "# calculate accuracy on training and test set using range of parameter values\n",
    "train_scores, test_scores = validation_curve(\n",
    "    RandomForestClassifier(),\n",
    "    features,\n",
    "    target,\n",
    "    param_name=\"n_estimators\",\n",
    "    param_range=param_range,\n",
    "    cv=3,\n",
    "    scoring=\"accuracy\",\n",
    "    n_jobs=-1)\n",
    "\n",
    "# calculate mean and standard deviation for training set scores\n",
    "train_mean = np.mean(train_scores, axis=1)\n",
    "train_std = np.std(train_scores, axis=1)\n",
    "\n",
    "# calculate mean and standard deviation for test set scores\n",
    "test_mean = np.mean(test_scores, axis=1)\n",
    "test_std = np.std(test_scores, axis=1)\n",
    "\n",
    "# plot mean accuracy score for training and test sets\n",
    "plt.plot(param_range, train_mean, label=\"Training score\", color=\"black\")\n",
    "plt.plot(param_range, test_mean, label=\"Cross-validation score\", color=\"dimgrey\")\n",
    "\n",
    "# plt accuracy bands for training and test sets\n",
    "plt.fill_between(param_range, train_mean - train_std, train_mean+train_std, color=\"gray\")\n",
    "plt.fill_between(param_range, test_mean - test_std, test_mean+test_std, color=\"gainsboro\")\n",
    "\n",
    "# create plot\n",
    "plt.title(\"Validation Curve With Random Fores\")\n",
    "plt.xlabel(\"Number Of Trees\")\n",
    "plt.ylabel(\"Accuracy Score\")\n",
    "plt.tight_layout()\n",
    "plt.legend(loc=\"best\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Discussion\n",
    "Most training algorithms contain hyperparameters that must be chosen before the training process begins.\n",
    "\n",
    "For example, a random forest classifier creates a \"forest\" of decision trees, each ofo which voteso n the predicted classo f an observaiton. One hyperparameter in random forest classifiers is the number of tress in the forest. Most often hyperparameter values are selcted during model selection (see Chapter 12). However, it is occasionally useful to visualize how model performance changes as the hyperparameters changes. In our solution, we plot the changes in accuracy for a random forest classifier for the training set and during cross-validation as the number of trees increases. When we have a small number of trees, both the training and cross-validation score are low, suggesting the model is underfitted. As the number of trees increases to 250, the accuracy of both levels off, suggesting there is probably not much value in the computation cost of traingin a massive forest.\n",
    "\n",
    "In sci-kit-learn, we can calculate the validation curve using `validation_curve`, which contains three important parameters:\n",
    "* `param_name` is the name of the hyperparameter to vary\n",
    "* `param_range` is the value of the hyperparmeter to use\n",
    "* `scoring` is the evaluation metric used to judge the model.\n",
    "\n",
    "#### See Also\n",
    "* scikit-learn documentation: Validation Curve (http://scikit-learn.org/stable/modules/learning_curve.html#validation-curve)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:machine_learning_cookbook]",
   "language": "python",
   "name": "conda-env-machine_learning_cookbook-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
